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■/  Improving  education 
V"  Identifying  priorities 
V'  Guiding  decisions 
v'  Addressing  needs 
•/  Enhancing  communication 

Throughout  Alberta,  school  systems  are  increasingly  gathering  feedback  and 
perspectives  from  beneficiaries  and  stakeholders  of  the  learning  system.  By 
increasing  communication  and  listening  to  ideas  and  opinions,  school  systems 
who  conduct  survey  research  are  gaining  greater  insight  into  the  ideas,  attitudes, 
and  opinions  of  the  Albertans  they  serve.  By  conducting  satisfaction  research, 
education  providers  are  becoming  better  prepared  to  address  the  needs  and 
expectations  of  Alberta  students,  parents  and  teachers. 

This  reference  guide  has  been  designed  to  assist  education  providers  to  plan, 
design  and  implement  satisfaction  surveys.  There  are  a variety  of  methods  and 
techniques  that  can  be  employed  to  conduct  surveys.  All  have  merits  and 
drawbacks  to  consider  when  deciding  how  to  implement  a survey.  The 
information  presented  in  this  reference  guide  is  intended  to  help  those  who  are 
involved  in  satisfaction  surveys  within  Alberta's  learning  system. 

School  jurisdictions  and  schools  within  Alberta  are  required  to  develop  education 
plans  that  incorporate  performance  outcomes  and  measures.  Often,  these 
outcomes  and  measures  are  related  to  the  satisfaction  of  parents,  students,  staff 
and  other  community  members.  The  information  presented  in  this  reference 
guide  can  be  used  to  help  school  jurisdictions  and  schools  plan  and  implement 
surveys  to  measure  satisfaction  within  these  constituent  groups. 

It  is  worth  noting  that  Alberta  Learning  conducts  annual  satisfaction  surveys  with 
students,  parents  and  the  public.  The  questions  asked  in  these  surveys  can  be 
found  on  the  Internet  at 

http://www.learninq.qov.ab.ca/educationsvstem/planninq.asp. 

Several  school  authority  representatives  (10)  in  Alberta  participated  in  a survey 
that  provided  valuable  information  for  this  reference  guide.  Their  input  and 
suggestions  has  been  incorporated  in  this  guide  and  practical  examples  are 
highlighted.  HarGroup  Management  Consultants,  Inc.  is  grateful  to  these 
representatives  and  acknowledges  their  constructive  and  valuable  suggestions. 


Inquires  regarding  conducting  satisfaction  surveys  should  be  directed  to: 
Alberta  Learning 

Performance  Measurement  and  Reporting  Branch 

9th  Floor,  Commerce  Place 

10155-  102  Street 

Edmonton,  Alberta,  T5J  4L5 

E-mail:  measurement.learning@gov.ab.ca 
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Why  Measure  Satisfaction 


Satisfaction  research  can  enrich  and  enhance  decision  making  processes  by 
providing  various  stakeholders  (students,  parents,  teachers  and  citizens)  an 
opportunity  to  provide  their  perspectives  on  the  quality  of  Alberta's  learning 
system.  By  understanding  what  stakeholders  think  about  the  system  and  how  it 
serves  the  community,  education  providers  can  make  improvements  to  more 
effectively  address  their  priorities,  values  and  ideas. 

Education  providers  typically  have  many  opportunities  to  gather  feedback  from 
stakeholders  of  the  learning  system.  Jurisdiction  or  school  representatives  can 
speak  directly  to  students  in  school  hallways  or  at  student  union  meetings. 
Parents  can  provide  feedback  at  parent/teacher  meetings  or  other  organized 
events.  Teachers  can  express  their  opinions  at  curriculum  meetings  or  face-to- 
face  discussions  with  administrators.  Citizens  can  attend  school  board  meetings 
and  present  opinions  about  how  the  learning  system  is  perceived  in  the 
community.  All  of  these  feedback  mechanisms  provide  opportunities  for 
stakeholders  to  express  their  views  and  opinions,  and  help  jurisdictions  and 
schools  determine  priorities  to  meet  the  expectations  of  the  community.  A 
satisfaction  survey  is  another  tool  that  enables  jurisdictions  and  schools  to 
understand  stakeholder  perspectives.  The  advantage  of  a survey  is  that  it 
enables  measurement  of  the  perspectives  of  all  who  are  being  served  by  the 
learning  system. 

Most  school  boards  and  schools  within  Alberta's  learning  system  already  use 
satisfaction  surveys  to  address  and  understand  a variety  of  critical  issues.  For 
example,  satisfaction  surveys  are  used  to  plan  education  programs,  examine  use 
of  technology  in  schools,  and  measure  students'  safety  and  security  within  and 
outside  of  school.  Satisfaction  surveys  are  also  conducted  by  school  boards  and 
schools  to  address  the  specifications  of  the  Alberta  Government  Accountability 
Framework,  which  is  an  ongoing  process  that  enables  school  boards  to 
implement  continuous  improvement  initiatives  (Alberta  Learning  publishes  a 
Guide  for  School  Board  Planning  and  Results  Reporting). 

Satisfaction  surveys  also  help  school  boards  and  schools  to  track  and  analyze 
stakeholder  feedback  over  time,  thus  enabling  them  to  identify  and  understand 
changes  in  stakeholder  perspectives.  Some  school  boards  and  schools  conduct 
annual  satisfaction  surveys  with  various  stakeholder  groups.  Others  survey 
target  populations  at  regular  intervals  (e.g  students  from  Grades  4,  8 and  12  are 
surveyed  every  three  years).  Whichever  approach  is  used,  school  boards  and 
schools  can  analyze  the  data  over  time  to  identify  changes  in  perceptions, 
priorities  and  expectations. 


Who  Shouid  be  Surveyed 

School  boards  and  schools  in  Alberta  conduct  satisfaction  surveys  with  a variety 
of  populations.  Most  notable  are  the  direct  beneficiaries  of  the  learning  system 
such  as  students  and  parents  or  guardians  of  students.  Other  beneficiaries  can 
be  included  in  satisfaction  surveys,  such  as  employers  within  the  school 
jurisdiction  and  the  general  citizenry.  These  latter  stakeholder  groups  receive 
indirect  economic  and  social  benefits  from  the  learning  system  such  as  a skilled 
workforce,  citizenry  prepared  to  contribute  to  society,  etc. 


-1  - 


Introduction 


A Practical  Guide  to  Conducting  Surveys  within  Alberta's  Basic  Learning  System 


Service  providers  may  also  be  targeted  in  satisfaction  measurement  processes. 
Teachers,  administrators,  and  school  board  members  are  often  surveyed  about 
job  satisfaction,  education  delivery,  infrastructure  priorities,  use  of  technology, 
etc. 


Conducting  Satisfaction  Surveys  - 5 Phases 

The  survey  process  is  commonly  conducted  in  five  key  phases,  which  are 
broadly  presented  in  the  diagram  on  the  next  page.  All  phases  are  important  to 
the  overall  success  of  the  satisfaction  measurement  process.  Each  survey 
project  undertaken  in  Alberta's  learning  system  will  be  distinct  and  may  include 
all  or  some  of  the  issues  presented  in  the  diagram,  but  most  will  follow  this 
framework. 

The  first  two  phases  involve  planning  the  survey  process  and  designing  the  tools 
and  instruments  that  will  be  used  in  the  survey.  These  phases  of  the  process 
directly  impact  the  remaining  phases  and  should  be  given  careful  consideration. 
For  example,  if  a survey  objective  is  not  identified  in  the  first  phase  (and 
subsequently  questions  are  not  designed  to  address  the  issue  in  the  second 
phase),  it  may  be  impossible  to  address  the  issue  once  the  third  phase 
(administration  of  the  survey)  is  complete. 

The  third  and  fourth  phases  encompass  conducting  the  survey  with  stakeholders, 
analyzing  the  results,  and  communicating  the  results  to  interested  individuals  or 
groups.  The  logistical  steps  in  conducting  the  survey  will  depend  on  the  method 
of  data  collection  selected  for  the  survey.  For  example,  in  a web-based  survey, 
respondents  may  be  invited  to  participate  in  the  survey  by  receiving  an  e-mail 
invitation  or  receiving  a letter  that  invites  them  to  participate.  Conversely,  in  a 
telephone  survey  interviewers  contact  respondents  and  verbally  invite  them  to 
participate  in  the  survey. 

Survey  data  will  also  need  to  be  organized  and  assembled  for  analysis  such  as 
recoding  and  collapsing  data  into  categories,  preparing  data  tables,  and  using 
subgroup  analysis  to  determine  whether  different  respondent  groups  have  similar 
or  distinct  opinions  about  issues.  The  survey  findings  should  be  organized  and 
presented  in  a report  that  interested  individuals  or  groups  can  use. 

In  the  final  phase,  education  providers  are  encouraged  to  develop  and  implement 
initiatives,  or  adjust  existing  policies  and  programs,  based  on  the  survey  results. 
In  some  cases,  the  survey  results  may  suggest  that  the  school  board  or  school  is 
addressing  the  expectations  and  priorities  of  stakeholders.  Alternatively,  the 
results  may  identify  specific  issues  that  need  to  be  addressed  by  the  school 
board  and  the  school. 

As  a final  task,  jurisdictions  and  schools  should  consider  the  successes  and 
challenges  associated  with  the  survey  process  and  identify  ways  that  may 
improve  future  surveys. 
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Figure  1 - Phases  of  a Satisfaction  Survey 


Plan 


d Develop  a description  of  the  satisfaction  survey  project 

m Define  survey  population/s  (e.g.  students,  parents,  teachers  or 
citizens) 

Identify  key  research  questions  or  objectives 
Establish  project  budget  and  timelines 
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Design 


□ Determine  data  collection  technique(s) 

Cl  Determine  sampling  method  and  size 

□ Design  survey  tools  and  instruments  (e.g.  questionnaire) 
d Pre-test  survey  tools  and  instruments 


Administer 

O Prepare  instruments,  staff  and  equipment  for  survey  administration 
O Coiiect  data  from  respondents 
□ Prepare  data  for  analysis 
d Analyze  data 


Communicate 

d Identify  stakeholder  groups  for  reporting  purposes 
d Determine  methods  to  report  survey  results 
d Prepare  survey  results  report 
d Communicate  results  to  relevant  stakeholders 


Follow-up  Actions 

d Develop  and  implement  follow-up  actions  to  address  survey  results 
d Gather  additional  feedback  from  stakeholders 
d Evaluate  the  successes  and  challenges  of  the  survey  project 
d Plan  future  satisfaction  measurement  initiatives 


/ 
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A Checklist  for  Survey  Projects... 


The  following  checklist  provides  a broad  overview  of  elements  that  should  be 
considered  before,  during  and  at  the  end  of  survey  research  projects.  Each 
project  will  be  distinct  and  may  include  all  or  some  of  the  issues  presented  in  the 
checklist. 


Getting 
started ... 


During 

implementation... 


□ Has  the  business  need  for  the  survey  research  been  established  and  articulated  (e.g. 
how  does  the  survey  relate  to  an  education  plan)? 

□ Has  a Survey  Plan  been  developed  (survey  description,  objectives,  target 
populations,  methodology,  use  of  external  contractors,  timelines,  deliverables,  budget 
requirements,  communications,  reporting,  and  use  of  results)? 

□ Does  the  survey  project  have  appropriate  approvals  (e.g.  School  Board, 
Superintendents,  etc)? 

□ Has  a Request  for  Proposal  been  developed  within  established  guidelines  and  with 
relevant  project  information  and  specifications  (to  distribute  to  external  contractors)? 

□ How  will  external  contractors  be  selected  (open  competition,  single  source,  or 
selected  competition)? 

□ Have  proposals  from  external  contractors  been  rigorously  evaluated  and  approved? 

□ Has  a formal  contract  been  signed  by  external  contractors  and  jurisdiction  or  school 
representatives? 

□ Are  data  collection  methods  appropriate  for  the  target  populations  and  project 
circumstances? 

□ Have  staff  who  deal  with  the  public  been  informed  about  the  survey  so  that  they  can 
answer  telephone  inquiries  about  it? 

□ Has  the  sample  plan  been  given  adequate  consideration? 

□ Have  the  survey  instruments  been  designed  effectively  and  field  tested? 

□ Have  quality  control  measures  been  established  and  employed  in  the  survey 
administration? 


Project 
completion  ... 


□ Are  the  specifications  of  the  Freedom  of  Information  and  Protection  of  Privacy  Act 
being  maintained? 

□ Have  the  survey  data  been  adequately  analyzed? 

□ Does  the  survey  report  identify  the  business  need,  describe  the  methodology,  provide 
a respondent  profile  and  present  factual  findings  obtained  in  the  research? 

□ Has  the  project  manager  adequately  evaluated  the  survey  research  project, 
identifying  strengths  and  weaknesses  and  suggesting  recommendations  for  future 
research? 


□ Have  the  deliverables  been  approved  by  the  person/s  responsible  for  the  survey  or 
senior  administration? 


□ Have  the  results  of  the  research  been  communicated  to  the  appropriate  stakeholders 
and  constituents  (both  internal  and  external)? 

□ Have  the  survey  results  (raw  data  and  reports)  been  sent  or  submitted  for  data 
storage  or  external  distribution? 

□ Have  the  survey  report  and  data  been  appropriately  warehoused  and  stored  within 
Alberta  Learning  protocols  and  Freedom  of  Information  and  Protection  of  Privacy  Act 
specifications? 
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Planning  a Satisfaction  Survey 


Every  survey  project  should  have  a plan  that  presents  key  information  about  the 
project.  The  plan  will  help  those  engaged  in  survey  research  to  organize  the 
tasks  required  to  successfully  design  and  execute  the  satisfaction  measurement 
process.  It  will  also  help  to  inform  others  within  the  school  board  or  school  about 
the  satisfaction  survey,  its  purpose,  the  process  and  how  the  results  might 
contribute  to  the  overall  decision  making  process  or  education  plan  of  the  school 
board  or  school.  As  well,  a documented  survey  plan  can  facilitate  continuity  and 
clarity  if  the  survey  is  conducted  on  an  annual  basis. 

The  survey  plan  should  show  a clear  relationship  between  the  survey  results  and 
the  decisions  that  will  be  made,  the  information  need,  the  methodology  to  be 
implemented,  and  the  reporting. 


□ Develop  a description  of  the  satisfaction  survey  project 


The  description 


should  portray  the 
intent,  context,  and 
scope  of  the 
satisfaction  survey. 


When  preparing  a description  of  the  satisfaction  survey,  those  involved  in  the 
survey  should  organize  basic  ideas  and  concepts  for  what  the  survey  is  about, 
who  is  involved  and  when  it  will  be  conducted.  A description  of  a satisfaction 
survey  will  typically  include  the  following: 


• Identify  the  sponsor  of  the  survey  (for  whom  the  survey  is  being 
conducted  - e.g.  school  board,  school,  etc.). 

• Identify  the  information  need  (why  is  the  survey  being  done  and  how  the 
results  will  support  planning/decision  cycles). 


• Determine  the  issues  that  need  to  be  resolved. 


• Set  priorities  for  issues  that  will  be  addressed  in  the  survey. 

• Identify  an  individual  within  the  organization  as  the  primary  contact  for 
the  survey.  This  individual  should  be  accessible  to  respondents  who 
may  have  questions  about  the  satisfaction  survey. 


in  Alberta  use 
outside  assistance 
to  help  with  all  or 
parts  of  the  survey 
project 

J 


Determine  the  project  deliverables  - data  tables,  written  reports, 
presentation  formats  (web-based,  paper,  or  electronic  files),  etc. 

Determine  whether  outside  assistance  is  needed  to  support  the  survey 
process  (consultants,  research  firms,  technology  contractors,  printing 
companies,  etc.).  See  Appendix  A for  information  about  contracting 
outside  consultants. 
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□ Define  Survey  Population 

The  survey  population  represents  the  complete  group  of  persons  (can  also  be 
objects,  businesses,  units,  etc.)  to  which  the  survey  results  will  be  generalized. 


Ten  Alberta  school 
authorities  were 
asked  which  grade 
levels  they  survey 
in  their  satisfaction 
projects.  Most 
surveyed  three 
grade  levels  (e.g. 
grades  4,  8 and  12) 
in  any  given  year. 
Few  school  boards 
survey  students  in 
ECS  or  grades  1,  2, 
and  3. 


Identify  target  populations  for  the  research.  In  the  context  of  the  learning 
system,  these  may  include: 

• Students 

• Parents 

• Teachers 

• Administrative  staff 

• Custodial  staff 

• School  Board  Members 

• Superintendents 

• Stakeholders 

• Citizens 

• Employers 

The  overall  satisfaction  survey  project  may  involve  all  or  some  of  the 
target  populations  identified  above. 

Determine  or  estimate  the  size  of  the  target  populations.  The  size  of  the 
population  will  be  needed  if  a sample  of  the  population  is  contacted  to 
participate  in  the  survey. 

Identify  factors  or  characteristics  of  the  target  population  that  might  limit 
the  ability  to  contact  members  of  the  population  (e.g.  lack  of  direct 
contact  information  for  parents,  vacation  schedules,  access  to 
computers,  etc.),  and  address  them  to  minimize  bias  in  the  survey 
results. 


□ Identify  key  research  questions  or  objectives 


are  statements  that 


clearly  describe 
what  you  want  to 
learn  or  understand 
from  the  survey 
results. 

V J 


Establish  a set  of  key  research  questions  or  survey  objectives  to  specify  the 

purposes  for  which  information  is  required. 

• Identify  the  issues,  problems  and  hypotheses  to  be  tested  in  the 
satisfaction  survey. 

• Ensure  that  potential  users  (decision  makers,  staff,  etc.)  can  understand 
what  issues  the  survey  research  addresses. 

• Develop  survey  objectives  in  partnership  with  decision  makers  and 
stakeholders. 

• Ensure  that  objectives  are  clear  and  concise. 

• The  final  results  of  the  satisfaction  survey  should  address  the  survey 
objectives  that  are  established. 
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□ Establish  Project  Budget  and  Timelines 

• Determine  the  timelines  for  the  project  including  milestones  for  key  phases 
and  the  completion  deadline.  The  following  is  a simple  illustration  of  a survey 
project  schedule  showing  tasks  and  timelines.  A survey  project  may  require 
more  details  to  identify  specific  tasks  and  responsibilities  for  various 
individuals  or  staff  members  involved. 


Survey  Project  Schedule: 

Week 

Task: 

1 I 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

Prepare  survey  objectives  and  budgets 

Design  survey  methodology  and  sample  plan 

Design  survey  instruments 

Pre-test  survey  instruments 

Prepare  and  send  out  survey  questionnaires 

Respondents  complete  and  return  questionnaires 

Clean  and  analyze  data 

Prepare  reports 

Communicate  results  to  relevant  stakeholders 

Develop  action  plans  to  address  survey  results 

Key  Milestones:  ❖ 


• In  many  cases,  the  survey  project  may  not  be  conducted  in  sequential  or 
linear  steps.  Survey  administrators  should  anticipate  when  various  activities 
might  be  undertaken  within  a survey  project  and  plan  appropriate  completion 
dates  for  each  activity.  For  example,  programming  of  scanning  technology 
might  occur  during  the  data  collection  phase  so  that  completed 
questionnaires  can  be  scanned  immediately  following  the  data  collection 
deadline. 


• Use  local 
specialists  for 
data  collection 


• Assign  clerical 
staff  to  perform 
non-technical, 
routine  tasks 

• Refer  to  existing 
survey 
instruments 

• Borrow  whenever 
possible 

(equipment,  staff, 
materials  and 
supplies). 

J 


When  estimating  survey  project  costs,  schools  and  school  boards  should 

consider  both  out-of-pocket  financial  costs  and  non-dollar  or  indirect  costs. 

Some  of  the  survey  costs  that  estimators  might  need  to  consider  include: 

• Internal  staff  - the  time  and  wages  of  internal  staff  to  organize  and 
implement  the  survey  project. 

• Travel  - internal  staff  may  be  required  to  travel  for  the  survey  project 
(e.g.  visiting  schools,  attending  meetings,  etc.). 

• Printing  - preparation  of  paper  questionnaires,  reports,  and  other  related 
materials. 

• Communications  - may  include  printing,  postage,  telephone  rentals  and 
long-distance,  couriers,  presentations,  etc. 

• Data  capture  and  processing  - costs  may  be  incurred  from  purchasing 
specialized  computer  software  (or  contracting  data  capture  and 
processing  to  outside  organizations). 

• Supplies  and  equipment  - basic  supplies  such  as  pencils,  pens,  paper, 
and  envelopes  may  be  needed.  Special  equipment  such  as  computers, 
projectors,  easels,  etc.  may  be  needed  to  implement  the  survey  and 
present  the  results. 

• Outside  consultants  - all  or  part  of  a satisfaction  survey  project  may  be 
contracted  to  outside  consultants.  Consultants  commonly  charge  rates 
on  a per  diem  basis  or  as  a fixed  rate  for  completing  a contracted 
service.  The  next  section  of  this  manual  examines  issues  related  to 
engaging  outside  consultants. 
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□ Contracting  Outside  Contractors/Consultants 

Outside  research  contractors/consultants  are  sometimes  engaged  to  assist  in 

designing  and  executing  a survey  project,  or  conducting  a specific  component  of 

the  project. 

There  are  typically  three  methods  of  sourcing  and  selecting  outside  consultants: 

• Open  Competition  by  Advertisement  - Service  requirements  are  advertised  to 
the  general  public  and  all  interested  parties  are  invited  to  submit  a proposal 
(e.g.  advertisements  in  newspapers).  This  method  can  often  result  in  a large 
number  of  proposals  being  submitted  for  review. 

• Single  Source  Selection  - Only  one  contractor/consultant  is  invited  to  submit 
a proposal  for  the  provision  of  services.  Generally,  organizations  should 
have  a compelling  reason  for  using  this  selection  process  (e.g.  only 
available  supplier,  limited  budget,  etc.).  It  is  important  to  ensure  that  the 
contractor/consultant  has  appropriate  qualifications  and  experience  to 
conduct  the  project,  if  this  method  is  chosen. 

• Select  Competition  by  Resource  List  - Specified  contractors/consultants  are 
invited  to  respond  to  a Request  for  Proposal.  At  least  3 to  5 
contractors/consultants  should  be  invited  to  respond.  A resource  list  can  be 
developed  by  contacting  other  jurisdictions  to  determine  which  survey 
research  firms  they  use,  or  by  contacting  Alberta  Learning.  It  should  be 
noted  that  some  contractors  may  not  submit  proposals  for  projects;  therefore 
organizations  may  want  to  invite  more  than  3 contractors  to  obtain  at  least  3 
proposals  for  consideration. 


/.  Request  for  Proposals 

Solicitation  for  services  can  be  conducted  through  a Request  for  Proposal  (RFP) 
process.  A RFP  is  a solicitation  method  used  to  seek  survey  project 
management  plans  and  proposed  costs  from  external  contractors. 

Develop  Request  For  Proposals 

A Request  for  Proposals  (RFP)  is  a document  used  to  notify  potential  contractors 
that  a project  is  about  to  be  initiated,  and  the  organization  is  interested  in 
receiving  proposals,  or  bids,  from  interested  external  contractors.  The  RFP 
usually  provides  requirements  and  specifications  about  the  survey  project  and 
requirements  of  the  external  contractors  in  conducting  services. 

The  RFP  provides  potential  contractors  with  information  needed  to  prepare  and 
submit  a proposal  to  provide  research  services.  Information  contained  in  the 
RFP  will  generally  include  contact  information,  specifications  regarding  the 
proposals  submitted,  and  some  background  information  regarding  the  project. 
Contractors  will  review  the  RFP  to  determine  whether  or  not  they  feel  they  can 
take  on  the  project,  and  whether  or  not  to  submit  a proposal  to  offer  services. 
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A Request  for  Proposal  (RFP)  should  include: 

□ Project  background,  information  and  description. 

□ Survey  objectives. 

n Use  of  survey  results, 
n Technical  specifications  of  the  survey. 

□ Timing  of  the  project. 

□ Project  deliverables  and  ownership  of  results. 

□ Proposal  requirements. 

n Proposal  evaluation,  contract  negotiations  and  award  specifications. 

□ Freedom  of  Information  and  Protection  of  Privacy  Act  specifications  (see 
Section  4.3). 

□ Response  instructions. 

n Period  of  proposal  commitment  (e.g.  30  days  following  the  submission  date). 

The  RFP  may  include  attachments  such  as  detailed  survey  plans,  previous 
years’  questionnaires/reports,  draft  survey  instruments,  resources  available  for 
the  project  and  other  materials. 
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a.  Proposal(s)  Evaluation  and  Contract  Development 

Organizations  and/or  satisfaction  survey  committees  should  develop  factors  to 
rate  proposals,  assign  weights  to  criteria,  and  finalize  evaluation  forms.  In  the 
case  of  a satisfaction  survey  committee,  members  usually  review  the  proposals 
individually  and  then  meet  to  discuss  and  arrive  at  a consensus  rating  for  each 
proposal. 


Evaluate  Proposals  and  Select  External  Contractor 

There  are  a number  of  issues  to  consider  when  assessing  the  competence  and 
suitability  of  an  outside  contractor/consultant  for  a specific  satisfaction  survey 
project.  The  following  list  provides  a basic  checklist  for  selecting  an  outside 
contractor: 

□ Does  the  proposal  exhibit  an  understanding  of  the  project  and  its 
requirements? 

n Is  the  proposed  methodology  sound? 

• Does  the  proposal  provide  a high  probability  of  accomplishing  the  study 
objectives? 

• Is  the  accuracy  of  the  data  ensured? 

• Is  the  reliability  of  the  data  ensured? 

• Is  the  approach/methodology  based  on  sound  research  design 
principles? 

• Is  there  enough  flexibility  to  ensure  success  if  project  parameters 
change? 

□ Are  project  timelines  and  costs  reasonable  and  within  specifications? 

□ Does  the  contractor  have  the  technical  expertise  and  experience  to  fulfill  the 
project? 

□ Does  the  contractor  have  the  resources  to  complete  the  project? 

□ Other  criteria  may  also  be  needed  depending  on  specifications  of  the  project. 

A template  and  example  of  an  evaluation  form  are  presented  in  Appendix  A. 

The  successful  external  contractor  should  be  informed  of  the  outcome  of  the 
selection  process  and  offered  a contract  for  service.  Unsuccessful  contractors 
should  be  notified  that  their  proposals  were  not  selected  for  the  project. 
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Designing  the  Survey 

There  is  no  one  best  method  to  use  when  conducting  satisfaction  surveys.  What 
may  work  best  for  one  school  board  or  school  may  not  be  the  best  approach  for 
another.  Selecting  methods  to  conduct  a satisfaction  survey  will  depend  on 
several  factors  such  as  the  type  of  information  that  is  needed  from  the  survey, 
the  targeted  population,  ease  of  contacting  respondents,  and  the  financial  and 
resource  costs  to  carry  out  a survey. 

This  section  of  the  manual  provides  organizations  with  guidelines  and 
specifications  to  consider  when  deciding  which  survey  design  methods  to 
employ. 

n Determine  data  collection  techniaue(s) 

In  choosing  an  optimal  data  collection  method  for  the  survey,  the  nature  and 
requirements  of  the  information  should  be  considered.  The  survey  population  will 
also  influence  the  data  collection  method  chosen  (how  can  the  population  be 
contacted  and  surveyed).  It  is  important  to  consider  these  factors  in  detail,  as  the 
data  collection  method  can  substantially  impact  the  quality  of  information 
obtained  by  the  survey. 

The  types  of  data  collection  methods  commonly  used  by  school  boards  and 
schools  within  Alberta's  learning  system  include  (presented  in  order  of  frequency 
of  use  by  school  boards  and  schools): 


Commonly  used  to  survey: 

• Students 

• Parents 

• Teachers 


Mail  survey  - (a  self-administered  survey)  questionnaires  are  sent  to 
respondents  through  the  mail  (or  some  other  delivery  method)  and  asked  to 
complete  the  form  and  return  it.  There  are  variations  of  the  mail  survey 
approach  that  might  not  use  postal  mail.  For  example,  teachers  could  provide 
students  with  a questionnaire  to  complete  in  class.  The  students  complete  the 
questionnaire  and  return  it  to  the  teacher  in  a sealed  envelope.  Another 
example  might  involve  students  taking  questionnaires  home  for  their  parents 
to  complete.  After  the  parents  have  completed  the  questionnaire,  they  put  it  in 
a sealed  envelope  and  their  children  take  it  back  to  the  school  to  give  to  the 
teacher  (or  return  to  some  other  drop  off  system).  In  interviews  with  Alberta 
school  authorities,  the  mail  survey  technique  was  the  most  common  technique 
used  to  survey  students,  parents  and  teachers. 


Becoming  a commonly 
used  survey  method: 

• Students 

• Teachers 

• Parents 


Web-based  survey  - (a  self-administered  survey)  respondents  are  asked  to 
interact  with  a computer  and  enter  their  responses  in  the  questionnaire  form 
by  using  a keyboard  or  touching  a computer  screen.  Some  schools  have 
used  this  method  when  surveying  students,  teachers  and  parents.  Schools 
have  organized  time  schedules  so  that  students  can  use  computer  labs  to 
access  computers  and  complete  the  survey.  Teachers  have  been  invited  to 
complete  questionnaires  at  a computer  terminal  available  in  the  school.  Some 
schools  have  sent  out  questionnaires  to  parents  with  students,  but  offered 
parents  the  option  to  complete  the  questionnaire  on-line  (in  most  cases,  this 
has  had  limited  success).  Some  organizations  are  employing  innovative 
approaches  to  encourage  parents  to  participate  in  satisfaction  surveys  using 
web-based  questionnaires.  For  instance,  parents  are  encouraged  to  visit  a 
school's  computer  lab  prior  to  the  annual  parent/teacher  interviews  to 
complete  a questionnaire  on-line. 


Design 
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Telephone  survey  - (an  interviewer  administered  survey)  trained  interviewers 
contact  respondents  by  telephone  and  administer  the  interview.  Interviewers 
ask  respondents  questions  and  record  responses  on  paper  or  computer-aided 
telephone  interview  forms.  The  telephone  survey  has  been  used  to  conduct 
satisfaction  surveys  with  parents,  local  employers  and  the  public.  Typically, 
this  survey  technique  has  not  been  employed  with  students  or  teachers 
because  these  groups  are  mostly  captive  audiences  within  the  learning 
system  and  can  be  efficiently  interviewed  through  the  mail  or  web-based 
survey  techniques  (as  presented  above). 

In-person  survey  - (an  interviewer  administered  survey)  trained  interviewers 
administer  the  interview  in-person  at  respondents'  homes,  offices,  schools, 
etc.  The  interviewer  is  responsible  for  asking  questions  and  recording 
responses  on  paper  or  computer-aided  interview  forms.  This  survey 
technique  is  not  commonly  used  to  conduct  satisfaction  surveys  within 
Alberta's  learning  system.  Yet,  it  may  present  some  advantages  when 
attempting  to  administer  complex  questionnaires  to  respondents  or 
interviewing  younger  students  for  a satisfaction  survey  (e.g.  grades  1 to  3) 


In  some  satisfaction  survey  projects,  combinations  of  the  data  collection  methods 
presented  above  have  been  employed  (e.g.  a mail  survey  conducted  with 
parents  and  web-based  surveys  with  students  and  teachers). 


The  table  on  the  following  page  presents  data  collection  methods  that  are 
commonly  used  to  conduct  satisfaction  surveys  in  Alberta's  learning  system, 
along  with  considerations  for  applying  each  technique. 

Another  consideration  for  deciding  upon  a data  collection  method  is  the  survey 
sampling  method  and  sample  size.  The  next  section  of  this  manual  examines 
this  component  of  survey  design. 


Design 


-12- 


A Practical  Guide  to  Conducting  Surveys  within  Alberta's  Basic  Learning  System 


Considerations  for  Applying  Survey  Techniques 


Mail  Survey 

(Self- 

Administered) 


Process 


A questionnaire  is 
sent  to 

respondents  to 
complete  and 
return  through 
mail. 


Advantages 


Complete  at  leisure 
Detailed  response 
Interviewer  bias  removed 
Can  use  visual  stimuli 

Potential  for  lengthy 
questionnaires 

Respondent  perception  of 
anonymity  is  moderate 


Disadvantages 


Lack  of  control  over 
verbatim  responses 

Challenges  with 
sequencing 

Slow  turn-around 

In  some  cases,  difficult  to 
encourage  response  (e.g. 
parents) 


Response  Rates 


Low  to  moderate 


• Students  - 
moderate  to  high 

• Parents  - low  to 
moderate 

• Teachers  - 
moderate  to  high 


Web-based 

Surveys 

(Self- 

Administered) 


A surveying 
method  in  which 
the  Internet  (or  an 
intranet)  is  used  to 
collect  data. 
Questionnaires 
are  distributed  to 
respondents  using 
electronic  mail,  or 
respondents  are 
asked  to  visit  a 
website  to 
complete  an  on- 
screen 

questionnaire. 


Complete  at  leisure 
Detailed  response 
Interviewer  bias  removed 
Can  use  visual  stimuli 
Quick  turn-around 
Inexpensive  to  add  sample 

Verbatim  responses  tend  to  be 
more  detailed 

Ability  for  complex  skips  and 
rotations 


• Potential  challenges  with 
representative  samples 
(respondents  must  have 
access  to  Internet) 

• Lack  of  control  over 
verbatim  responses 

• Challenges  with 
sequencing 

• Lower  tolerance  for 
questionnaire  length 

• Need  for  de-bugging 

• Potential  for  crashes 

• In  some  cases,  difficult  to 
encourage  response  (e.g. 
parents) 

• Respondent  perception  of 
anonymity  may  be  low 


Low 


Students  - 
moderate  to  high 

Parents  - low 

Teachers  - 
moderate  to  high 


Telephone 

Survey 

(Interviewer 

Administered) 


These  surveys 
involve  one-on- 
one  questioning 
and  answering 
between  an 
interviewer  and 
respondent  by 
means  of  a 
telephone. 


Control  over  sequencing 
Relatively  fast-turn  around 

Able  to  control  samples  and 
quotas 

Can  access  widely  dispersed 
samples 

Use  of  computer  aided 
telephone  interviewing  (CATI) 
reduces  costs  with  automatic 
data  entry 

Typically,  interviewers  can 
encourage  participation 

Respondent  perception  of 
anonymity  is  moderate 


Potential  for  interviewer 
influence 

There  are  limits  to  what 
respondents  can  recall 
under  pressure 

May  be  less  convenient  for 
respondents 

Does  not  permit  visual 
stimuli 

Need  for  CATI  debugging 


Moderate 


Typically  high  rates 
of  return 


In  Person 
Survey 

(Interviewer 

Administered) 


A face-to-face 
meeting  between 
an  interviewer  and 
respondent,  which 
may  be  conducted 
in  an  office,  home, 
school,  etc. 


Control  over  sequencing 
Relatively  fast-turn  around 

Potentially  able  to  conduct 
longer  interviews 

Potential  for  lengthy 
questionnaires 

Typically,  interviewers  can 
encourage  participation 

Respondent  perception  of 
anonymity  is  moderate 


Pressure  on  respondent  can 
influence  ability  to  recall 

May  be  less  convenient  for 
respondents 

Reduces  ability  to  supervise 

Greatest  potential  for 
interviewer  influence 

Can  be  affected  by  weather 


High 


Typically  high  rates 
of  return 
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□ 


A small  survey  of 
ten  Alberta  School 
authorities  revealed 
that  most  adopted 
census  approaches 
for  conducting 
satisfaction 
surveys.  For 
example,  all 
students  in  grades 
4,  8 and  12,  and 
their  parents,  were 
provided  an 
opportunity  to 
participate  in  the 
satisfaction  survey. 
As  well,  all  teachers 
and  administrative 
staff  within  the 
authorities  could 
participate  in  the 
survey. 

The  census 
approach,  however, 
might  be  less 
practical  when 
surveying  local 
employers  or  the 
general  citizenry. 


Determine  sample  method  and  size 

Satisfaction  surveys  may  be  conducted  on  an  entire  population  (i.e.  a census 
survey),  or  on  a sample  of  a population.  This  approach  might  be  used  when  the 
survey  population  is  small,  or  when  comprehensive  data  is  required.  Another 
reason  for  conducting  a census  is  to  allow  everyone  within  a population  the 
opportunity  to  provide  feedback.  Often  though,  satisfaction  surveys  are 
conducted  on  a portion  of  the  population  (i.e.  a sample).  This  approach  reduces 
costs  where  the  population  to  be  surveyed  is  large.  To  provide  reliable  data 
that  can  be  used  to  make  inferences  about  the  population  as  a whole,  it  is 
important  that  the  sample  survey  be  conducted  on  a portion  or  sample  of 
the  population  that  is  representative  of  the  whole. 

With  a census  survey  approach,  all  members  of  the  targeted  population  are 
offered  an  opportunity  to  participate  in  the  satisfaction  survey.  This  does  not 
mean  that  all  members  will  participate  (unless  legislated  by  law).  Rather,  each 
member  has  an  opportunity  respond  to  the  survey,  but  it  is  their  choice  as  to 
whether  they  will  participate  (at  least  in  the  context  of  conducting  surveys  within 
Alberta's  learning  system).  In  most  cases,  the  data  gathered  using  a census 
approach  will  result  in  a sample  of  the  population  (rather  than  a true  census 
where  100%  of  the  population  is  surveyed).  Where  substantially  less  than  100% 
of  the  population  responds  to  a census  survey,  those  responsible  for  the  survey 
analysis  should  check  for  characteristics  of  the  non-respondents  that  might  bias 
the  survey  results. 

Various  sampling  methods  can  be  employed  for  surveys  (see  following  page); 
however,  probability  samples  should  be  used  whenever  possible.  With 
probability  sampling,  each  member  within  a population  (or  sub-populations)  has  a 
known  probability  (e.g.  an  equal  opportunity)  of  being  selected  to  respond  to  the 
survey  and  inferences  to  the  population  can  usually  be  drawn  about  the 
population  from  the  responses  of  the  sample.  Non-probability  samples  may  be 
biased  because  of  the  way  respondents  are  selected  to  participate  in  the  survey. 
If  a sample  is  biased,  it  can  be  difficult  (if  not  impossible)  to  draw 
inferences  about  the  entire  population  from  the  data  that  are  gathered. 

Sample  size  is  also  an  important  factor  to  consider  in  the  sample  design. 
Essentially,  the  larger  the  sample,  the  more  certainty  the  survey  researchers  can 
have  that  the  data  gathered  in  the  survey  will  be  representative  of  the  opinions  of 
the  population.  The  smaller  the  sample,  the  more  likely  the  data  from  the  sample 
will  differ  significantly  from  the  opinions  of  the  population. 


The  relationship  between  sample  size  and  accuracy  of  findings  is  referred  to  as 
sampling  error  (or  margin  of  error)  and  is  a measure,  usually  estimated,  of  the 
extent  to  which  data  from  the  survey  sample  will  represent  the  entire  population. 
Media  often  report  survey  results  and  present  the  sampling  error  for  the  survey. 
For  example,  77%  of  respondents  were  satisfied,  plus  or  minus  5%.  The  ±5% 
represents  the  estimated  sampling  error  for  the  survey  data.  In  other  words,  the 
actual  number  of  satisfied  respondents  likely  lies  within  the  range  of  72%  and 
82%. 
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Sampling  Approaches  and  Methods 

Sampling  Approaches  Sampling  Methods 

Probability  Samples  - All  members  of  a 
population  have  a known  chance  of  being 
selected  into  the  sample 

Systematic  Sampling  - Starting  at  a random 
point  within  a list  of  population  members,  a 
constant  skip  interval  is  used  to  select  every 
other  sample  member. 

Simple  Random  Sampling  - A table  of 
random  numbers,  random  digit  dialing,  or  some 
other  random  selection  procedure  is  used  to 
ensure  that  each  member  of  a population  has 
the  same  chance  of  being  selected  into  the 
sample. 

Cluster  Sampling  - The  population  is  divided 
into  geographic  areas,  each  of  which  must  be 
very  similar  to  the  others.  A few  areas  can  be 
selected  to  conduct  a census  or  a sample  can 
be  drawn  from  a select  group  of  areas. 

Stratified  Sampling  - If  a population  is 
expected  to  have  a skewed  distribution  for  one 
or  more  distinguishing  factors,  subpopulations 
or  strata  can  be  identified  and  a simple  random 
sample  can  be  drawn  from  each  stratum. 
Weighting  procedures  may  be  reguired. 

Non-Probability  Samples  - The  chance  that 
members  of  a population  have  been  selected 
into  the  sample  is  unknown. 

Convenience  Sampling  - A "high-traffic"  area 
is  used  to  select  respondents  for  a sample  (e.g. 
a school  of  high  enrolment,  an  industrial  area 
of  a community,  etc.) 

Judgment  Sampling  - A researcher  uses 
his/her  own  judgment  to  identify  which 
respondents  will  be  in  the  sample. 

Referral  Sampling  - Respondents  are  asked 
for  names  of  others  like  themselves  who  might 
participate  in  a survey  (also  known  as  snowball 
sampling). 

Quota  Sampling  - Quotas  are  identified  for 
sub-populations  of  a population.  Respondents 
are  screened  to  determine  whether  they 
represent  the  parameters  of  the  guota. 

Sampling  error  is  estimated  within  a confidence  interval.  A confidence  interval 
indicates  the  probability  (e.g.  95  times  in  100)  that  the  true  value  lies  within  a 
specific  range. ^ It  is  common  for  survey  findings  to  rely  on  a 95%  confidence 
level,  but  higher  (99%)  and  lower  levels  (90%)  are  acceptable.  The  level  of 
confidence  selected  depends  on  the  degree  of  reliability  that  is  deemed 
acceptable  for  the  survey  findings.  Another  way  of  communicating  confidence 
intervals  is  to  say  "19  times  out  of  20,"  which  represents  a 95%  confidence  level. 

Another  point  to  consider  when  determining  sample  size  is  the  types  of  analyses 
that  will  be  undertaken.  For  example,  if  the  researcher  wants  to  compare  results 
among  various  schools  within  an  authority,  consideration  should  be  given  to 
increasing  the  overall  sample  size  to  facilitate  greater  confidence  in  findings  from 
the  comparative  analysis. 


^ Source:  How  to  Conduct  Customer  Surveys,  Institute  of  Citizen-Centred  Service,  2001. 
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The  following  table  provides  sample  sizes  for  various  sampling  errors  at  the  95% 
confidence  level.  For  instance,  if  a school  has  1 ,000  students,  approximately 
286  would  need  to  participate  in  the  satisfaction  survey  for  the  data  to  achieve  an 
estimated  ±5%  sampling  error  (margin  of  error)  within  a 95%  confidence  level. 


Sample  Size  Table  at  95%  Confidence  Interval 
(assuming  a probability  sample) 

Population  Size 

Sample  Size  Based  on  Estimated  Sampling  Error 
±3%  ±5%  ±7%  ±10% 

25 

24 

24 

22 

20 

50 

48 

44 

40 

33 

100 

92 

80 

67 

50 

200 

169 

133 

101 

67 

300 

236 

171 

121 

75 

400 

294 

200 

135 

80 

500 

345 

222 

145 

83 

600 

390 

240 

152 

86 

800 

465 

267 

163 

89 

1,000 

526 

286 

169 

91 

2,500 

769 

345 

189 

96 

5,000 

909 

370 

196 

98 

10,000 

1,000 

385 

200 

99 

25,000 

1,064 

394 

202 

o 

o 

T — 

100,000 

1,099 

398 

204 

100 

500,00  or  more 

1,110 

400 

204 

100 

1 Adapted  from  a sample  size  table  presented  in  How  Biq  Should  the  Sample  Be?  Statistics  Canada,  1993  | 

Many  school 
authorities  in  Alberta 
have  adopted  the 
census  approach  to 
survey  parents  in 
mail  surveys.  One  of 
the  reasons  for 
implementing  this 
approach  is  to 
compensate  for  high 
non-response  rates 
(between  50%  to  85%) 
typically  experienced 
in  mail  surveys  of 
population. 


Non-response  is  another  issue  to  consider  when  determining  sample  size.  In 
some  cases,  non-response  may  be  high  (e.g.  a mail  or  web-based  survey  of 
parents)  or  low  (a  web-based  survey  in  which  students  complete  the 
questionnaire  in  a class  lab).  This  factor  needs  to  be  considered  when 
determining  a sample  size  for  a survey. 

Sample  Frame  - Any  list,  database,  reference  material,  or  device  that  identifies, 
and  allows  access  to  members  of  a target  population  could  be  used  as  a sample 
frame  for  a satisfaction  survey.  Examples  of  a sample  frame  within  Alberta's 
learning  system  might  be  a list  of  parents  (with  addresses,  telephone  numbers  or 
e-mail  addresses),  a database  of  students'  e-mail  addresses,  and  telephone 
yellow  pages  that  list  companies  in  a local  area. 

The  sample  frame  used  for  a survey  should  be  as  comprehensive  as  possible. 
Any  member  information  missing  from  a sampling  frame  will  cause  sampling 
errors.  Researchers  may  need  to  update  or  improve  the  sample  frame  prior  to  a 
survey  to  minimize  the  potential  for  errors  in  the  sampling  process. 

In  some  cases  (e.g.  a telephone  survey),  a sample  frame  will  need  to  have 
significantly  higher  numbers  of  contacts  in  order  to  obtain  the  desired  sample 
size.  For  example,  a telephone  survey  may  require  a listing  of  contacts  that  is  3 
or  4 times  higher  than  the  targeted  sample  to  achieve  the  volume  of  completed 
responses  needed. 
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□ Design  survey  tools  and  instruments  (e.a.  questionnaire) 


Several  drafts  of  a 
questionnaire  may 
be  necessary  before 
the  instrument  is  in  a 
form  that  will 
effectively  address 
the  survey  objectives 
and  can  be 
administered 
efficiently  and 
effectively  with 
respondents. 


A questionnaire  is  a tool  comprised  of  a group  or  sequence  of  questions  that  are 
designed  to  elicit  information  about  one  or  more  subjects  from  a respondent.  A 
well-designed  questionnaire  should  collect  information  from  a respondent 
efficiently  and  effectively,  whether  it  is  interviewer-administered  or  self- 
administered  (i.e.  respondent  completed).  It  should  be  focused  to  achieve  the 
survey  objectives  set  out  in  the  survey  plan,  and  present  questions  that  are  easily 
understood  and  can  be  accurately  answered  by  respondents. 

Typically,  the  questionnaire  is  one  of  several  instruments  that  are  used  in  a 
survey.  Survey  instruments  involve  introduction  letters,  data  collection  forms, 
reminder  notices,  return-envelopes,  computer  software  programs,  etc.  These 
instruments  serve  numerous  functions  in  conducting  surveys,  for  instance: 

• Introduce  the  survey  to  the  respondents, 

• Establish  rapport  with  respondents, 

• Encourage  survey  participation, 

• Screen  respondents  to  systematically  identify  appropriate  subjects, 

• Gather  information,  and 

• Enable  capture  or  collection  of  the  data. 


The  composition  of  these  instruments  will  have  a major  impact  on  data  quality, 
respondent  relations  and  interviewer  performance  (where  appropriate).  The 
following  descriptions  present  the  composition  and  organization  of  the 
instruments  and  some  key  elements  of  questionnaire  development. 

Instrument  Composition  and  Organization 

• Introduction  - The  introduction  describes  the  purpose  of  the  survey, 
establishes  a rapport  with  respondents,  encourages  participation,  and 
screens  respondents  for  qualifications.  Ensure  that  the  value  of  providing 
information  is  made  clear  to  respondents  by  explaining  how  the  information 
will  be  used.  Respondents  should  also  be  assured  that  their  responses  are 
confidential  and  that  their  participation  is  voluntary.  The  following  elements 
should  be  present  in  the  introduction  section  of  questionnaires: 

^ Identification  of  survey  organization/sponsor. 

Purpose  of  the  survey. 

^ Explanation  of  respondent  selection. 

^ Request  for  participation/provide  incentive. 

^ Screening  of  respondent. 

• Identification  of  Survey  Sponsor  - In  the  context  of  conducting  satisfaction 
surveys  within  Alberta's  learning  system,  the  sponsor  (school  authority  or 
school)  should  be  identified  in  the  introduction  to  the  questionnaire.  A 
contact  person  should  also  be  available  to  respondents. 
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Several  schools  have 
used  symbols  as 
response  categories 
for  younger  students 
(grades  1,  2,  & 3)  to 
respond  to  surveys. 


e.g. 


Screening  process  - A process  that  systematically  selects  respondents  to 
interview  and  excludes  those  who  do  not  qualify.  The  process  can  also 
randomize  the  selection  of  respondents  (e.g.  "May  I speak  to  the  person  in 
your  household  whose  birthday  comes  next?"). 

Use  screening  questions  that  exclude  the  largest  proportions  of 
unqualified  respondents  first. 

^ Avoid  placing  sensitive  screening  questions  first. 

Use  screening  process  to  fill  quotas. 

Keep  track  of  call  record  for  estimates  or  weighting. 

^ Use  clear  quota/termination  instructions. 

Instructions  - In  some  cases,  respondents  will  be  presented  with  instructions 
to  participate  in  a survey  or  complete  a questionnaire.  Instructions  may  be 
presented  in  a separate  document  to  the  data  collection  form  (questionnaire), 
at  the  beginning  of  the  data  collection  form  or  for  individual  questions 
presented  in  the  data  collection  form.  These  instructions  should: 

Be  as  concise  as  possible. 

Avoid  emotionally  loaded  terms. 

Be  clear,  uncluttered,  and  appealing. 

^ Possibly  use  symbols  or  illustrations  (e.g.  Place  a check  in  the 
appropriate  box ...  e.g.0). 

Data  collection  forms  - Data  collection  forms  present  questions  that  address 
the  survey  objectives  and  question  responses  used  to  record  answers  on  the 
form.  These  forms  may  be  printed  on  paper,  or  presented  electronically. 
Another  component  of  data  collection  forms  is  the  equipment  or  materials 
that  might  be  used  to  capture  the  data  (e.g.  return-envelopes,  computers, 
hand-held  devices,  etc.). 


Basic  Questionnaire  Development 

Questionnaires  should  be  designed  to  address  the  following: 


• Present  relevant  questions  that  are  required  to  address  the  survey 
objectives.  In  other  words,  questions  should  focus  on  the  topic  of  the  survey. 

• Start  with  easy,  unthreatening,  but  pertinent  questions,  to  build  a rapport  with 
the  respondent.  Demographic  questions  should  only  be  present  at  the 
beginning  of  a questionnaire  when  they  are  being  used  for  screening 
purposes. 

• Definitions  should  be  clearly  stated  to  respondents. 

• Be  as  brief  as  possible. 

• When  switching  topics  within  a questionnaire,  use  a transitional  phrase  to 
allow  respondents  to  'switch'  their  thoughts. 
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• Use  filter  questions  to  let  respondents  avoid  sets  of  questions  that  do  not 
pertain  to  them. 

• Personal  information  gathered  within  questionnaires  should  comply  with 
specifications  presented  in  the  Freedom  of  Information  and  Protection  of 
Privacy  Act. 

• Question  structure,  wording  and  formatting  should  minimize  any  biasing  of 
responses.  At  the  very  least,  questions  should: 

V"  Address  a single  concept  only. 

V"  Not  use  double  negatives,  unfamiliar  words,  abbreviations,  acronyms, 
trade  jargon,  colloquialisms,  etc. 

Avoid  humor. 

Be  time  specific. 

Be  as  concise  as  possible. 
v"  Avoid  emotionally  loaded  terms. 
v"  Be  clear,  uncluttered,  and  appealing. 

Not  lead  respondents  to  provide  a specific  response. 

^ Avoid  implied  alternatives. 

Types  of  Questions 

There  are  three  main  types  of  questions  posed  to  respondents  in  questionnaires: 

• Open-ended  questions  - enable  respondents  to  answer  in  their  own  words. 
An  open  question  allows  respondents  to  interpret  the  meaning  of  the 
question  and  provide  an  answer  that  addresses  their  interpretation. 

e.g.  What  do  you  like  about  the  school  your  child  attends? 


Open-ended  questions  are  presented  in  questionnaires  for  several  reasons: 

To  provide  respondents  an  opportunity  for  self-expression 
To  allow  respondents  to  elaborate  on  complex  issues 
To  allow  a full  range  of  responses 
To  allow  for  clarification 
To  obtain  'natural'  wording 

To  gather  numerical  data  (e.g.  In  what  year  were  you  born?  19 ) 

To  add  variety  to  the  questionnaire 

There  are  challenges  with  open-ended  questions,  for  both  the  respondent 
and  the  researcher  that  is  conducting  the  satisfaction  survey.  For 
respondents,  open-ended  question  can  be  demanding  and  require  more 
thinking  than  closed  ended-questions.  As  well,  it  can  be  more  time 
consuming  for  the  respondent  to  answer  an  open-ended  question.  In  terms 
of  researchers  who  conduct  surveys,  it  can  be  time  consuming  and  difficult 
to  record  the  responses,  there  is  likely  a need  to  code  the  data  into 
categories,  and  it  can  be  difficult  to  analyze  and  interpret  the  data. 


Design 


-19- 


A Practical  Guide  to  Conducting  Surveys  within  Alberta's  Basic  Learning  System 


• Closed-ended  Questions  - respondents  are  offered  prescribed  answers  in 
which  they  can  choose.  Essentially,  respondents  are  restricted  to  choosing 
an  answer  or  response  option  that  is  specified  to  them. 


There  are  generally  three  types  of  closed-ended  questions  commonly  used 
in  satisfaction  surveys  within  Alberta's  learning  system,  including: 


Two-choice  or  multiple- 
choice  questions  are 
used  to  determine 
whether  one  alternative 
will  be  favored  over  the 
rest,  or  if  proportions  of 
the  population  tend  to 
prefer  various 
alternatives. 


A checklist  question  is 
used  when  researchers 
are  interested  in  how 
often  a particular 
response  option  is 
chosen  by  respondents 
or  the  frequency  by 
which  one  option  is 
chosen  over  others. 


• Two-choice  question  (only  two  choices  are  available  to  the  respondent): 

e.g.  Are  there  any  children  in  your  household  who  are  attending 
elementary,  junior  high  or  senior  high  school  in  Alberta? 

□ Yes  DUo 

• Multiple-choice  question  (more  than  two-choices): 
e.g.  Are  you  currently  enrolled  in  ... 

□ a public  school  □ a separate  school  □ a private  school 

• Checklist  question  (check  as  many  choices  as  apply): 

e.g.  What  subjects  have  you  studied  this  term  (select  all  that  apply) ... 

□ English  □ Math  □ Science  □ Social  Studies 

Closed  ended  questions  are  usually  easy  and  fast  for  respondents  to 
answer  and  are  easy  for  the  researcher  who  conducts  the  survey  to  code 
and  analyze.  Closed-ended  questions  can  take  effort,  on  the  part  of  the 
researcher,  to  develop  and  may  oversimplify  an  issue. 


• Scale-Response  Questions  - Response  scale  questions  are  instruments 
used  to  measure  phenomena  (e.g.  issues,  experiences,  perceptions,  etc.) 
that  cannot  be  easily  assessed  by  direct  means  (e.g.  observable  incidents  of 
behavior).  Respondents  are  asked  to  consider  their  response  or  answer  to  a 
question  based  on  a scale,  range  or  rank. 


There  are  numerous  types  of  scaled  response  questions.  The  following 
provides  some  examples  of  scale-response  questions  that  might  be  used  in  a 
satisfaction  survey  about  the  learning  system: 


Scaled  questions  are 
used  to  determine  a 
level  of  measurement 
about  an  issue,  topic  or 
subject.  Ranking 
questions  are  typically 
used  to  identify 
respondents'  favorability 
among  different  options. 


Do  you  agree  or  disagree  that  your  school  provides  a safe  environment? 

□ Agree  □ Disagree 

Overall,  how  satisfied  or  dissatisfied  are  you  with  the  quality  of  education 
you  receive  in  your  school?  Are  you... 

□ Very  satisfied  □ Satisfied  □ Dissatisfied  □ Very  Satisfied 

Please  rank  each  subject  in  terms  of  your  preference.  Place  a 'T  by  your 
first  choice,  a '2'  by  your  second  choice,  and  so  on. 


English  Math  Science  Social  Studies 
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The  Performance 
Measurement  and 
Reporting  Branch  of 
Alberta  Learning 
uses  a 4 point 
satisfaction  scale  for 
its  annual  parents, 
student,  and  teachers 
survey: 


e.g. 


Very  satisfied 
Satisfied 
Dissatisfied 
Very  satisfied. 


There  is  no  industry  standard  for  the  type  of  scale  questions  that  might  be 
used  in  each  and  every  situation.  Any  number  of  question  items  and  scale 
categories  may  be  created  depending  on  the  nature  of  the  issues  that  are 
being  investigated  in  the  survey  research. 

Overall,  when  selecting  a response  scale  question,  researchers  should 
consider  whether  the  question  (and  its  response  scale)  is: 

Easy  for  respondents  to  understand. 

Easy  for  researchers  to  interpret. 

Minimizing  response  bias. 

Easy  to  distinguish  between  point  intervals. 

^ Relevant  to  the  business  decision. 

Additional  issues  to  consider  when  developing  or  using  response  scales 
questions  include: 


• Regardless  of  the  number  of  categories  selected  in  the  response  scale, 
researchers  should  not  compare  or  perform  comparative  analysis  of 
dissimilar  and  unrelated  scales  (because  interval  relationships  among 
different  scales  cannot  be  correlated).  For  example,  the  results  of  a 4-point 
scale  should  not  be  compared  to  those  of  a 5-point  scale.  As  well,  a scale 
using  'satisfaction'  categories  should  not  be  compared  with  'agreement' 
categories. 


• Researchers  should  not  report  neutral  or  middle  category  responses  to  odd 
numbered  scales  as  affirmative  or  negative  opinions  or  perceptions,  unless 
an  unbalanced  scale  has  been  specifically  articulated  to  respondents 
(limitations  to  unbalanced  scales  are  presented  later  in  this  section). 


Do  Not  Report  Neutral  or  Middle  Category  as 
Affirmative  or  Negative  Opinion  or  Perception 


How  satisfied  or  dissatisfied 
are  you,  using  a scale  of: 

• Very  satisfied 

• Satisfied 

• Neither  satisfied  nor 
dissatisfied 

• Dissatisfied 

• Very  Dissatisfied 


How  satisfied  or 
dissatisfied  are  you,  using 
a scale  of: 

• 1 being  very  satisfied, 
and 

• 5 being  very  Dissatisfied 


Middle  Category  can  be 
Reported  As  Affirmative 

How  satisfied  or  dissatisfied  are 
you,  using  a scale  of: 

• Very  strongly  satisfied 

• Strongly  satisfied 

• Somewhat  satisfied 

• Somewhat  dissatisfied 

• Dissatisfied 
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• Researchers  should  enquire  with  other  researchers  and  Alberta  Learning  as 
to  response  scale  questions  being  used  in  similar  satisfaction  surveys  to 
enable  comparative  analysis  across  studies.  Typically,  Alberta  Learning 
researchers  have  used  balanced  (see  below)  or  equally  weighted  affirmative 
or  negative  4 or  5 point  scales  in  survey  research. 


Balanced  4-Point  Scale 

• Very  satisfied 

• Satisfied 

• Dissatisfied 

• Very  dissatisfied 


Balanced  5-Point  Scale 

• Very  satisfied 

• Satisfied 

• Neither  satisfied  nor  dissatisfied 

• Dissatisfied 

• Very  Dissatisfied 


The  following  examples  depict  scales  commonly  used  in  satisfaction  surveys: 

• Satisfaction  scale  - How  satisfied  or  dissatisfied  are  you  with  ...?  Please 
use  a scale  of  'very  satisfied',  'satisfied',  'dissatisfied',  and  'very  dissatisfied.' 
(add  'neither  satisfied  nor  dissatisfied'  added  as  a mid-point  for  a 5 point 
scale). 

• Performance  scale  - How  would  you  rate  ...?  Would  that  be  'very  good', 
'good',  'fair'  and  'poor'  (add  'excellent'  as  a starting  point  for  a 5 point  scale)? 

• Expectation  scale  - Compared  to  what  you  expected,  how  would  you  rate 
...?  Would  you  say  'much  better  than  expected',  'better  than  expected',  'about 
as  expected',  'worse  than  expected'  and  'much  worse  than  expected'. 

• Priority  scale  - How  important  is  ...  to  you?  Please  use  a scale  of  'not  at  all 
important',  not  important',  'important'  and  'very  important'. 

• Improvement  scale  - Indicate  the  amount  of  improvement,  if  any,  that  is 
needed?  Would  you  say  'none',  'slight',  'some',  'much'  or  'huge'. 

Alberta  Learning  conducts  annual  satisfaction  surveys  with  students,  parents  and 
teachers  in  the  province.  Schools  and  school  boards  involved  in  survey  research 
can  review  the  questionnaires  used  in  these  surveys  by  accessing  the  following 
web  pages: 

www.learninq.ab.ca/educationsvstem/satisfaction/default.asp 

Also,  Appendix  B presents  a four  step  process  for  researchers  to  use  when 
developing  scale  response  questions. 
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□ Pre-test  survey  tools  and  instruments 


It  can  be  helpful  to 
have  individuals  not 
directly  involved  in 
the  survey  project 
review  the 
questionnaire.  They 
may  find  issues 
(comprehension, 
logic  of  structure, 
clarity  of  instruction, 
etc.)  that  are  not 
readily  apparent  to 
the  researcher 
managing  the 
satisfaction  survey 
project 


Survey  tools  and  instruments  (including  questionnaires)  should  be  tested,  under 
field  conditions,  prior  to  implementation.  Essentially,  pre-testing  is  an  informal 
test  of  the  tools  and  instruments  usually  administered  in  the  same  manner  as 
planned  for  the  survey.  The  entire  questionnaire  can  be  tested,  or  a portion  of 
the  questionnaire  may  be  tested  (mainly  in  cases  where  most  of  the  questions 
have  been  asked  in  previous  surveys). 

Pre-tests  are  undertaken  to: 

^ Discover  vague  question  wording,  or  poor  ordering. 

Identify  errors  in  questionnaire  layout,  instruction,  or  sequencing. 

^ Identify  questions  that  respondents  are  unable  or  unwilling  to  answer. 

^ Develop  response  categories. 

Detect  bias  in  questions. 

Determine  suitability  of  the  questionnaire  for  measuring  concepts. 

Measure  interview  length  and  refusal  patterns. 

The  size  of  the  pre-test  sample  should  depend  on  the  specifications  of  the  project 
(e.g.  population  or  survey  sample  size,  questionnaire  previously  used  in  past 
surveys,  etc.).  The  average  pre-test  consists  of  approximately  10-15  completed 
interviews. 


Pilot  testing  may  also  be  necessary,  especially  for  large  and  complex  surveys. 
Pilot  testing  is  essentially  a "dress  rehearsal"  and  duplicates  the  final  survey 
design  on  a small  scale  from  beginning  to  end,  including  data  processing  and 
analysis.  Pilot  testing  provides  an  opportunity  to  refine  the  questionnaire,  as  well 
as  the  overall  survey  administration  process. 

It  may  be  necessary  to  test  the  survey  instruments  more  than  once. 
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Administering  the  Survey 


ni  Prepare  instruments,  staff  and  equipment  for  survey  administration 

Normally,  there  are  a variety  of  tasks  to  undertake  to  prepare  for  the  data 
collection  process.  Tasks  may  differ  depending  on  the  data  collection  technique 
chosen  for  the  survey. 

• Mail  surveys 


Some  school 
authorities  have 
partnered  or  shared 
resources  with  other 
authorities  to 
conduct  surveys. 


e.g.  sharing 
questionnaire 
scanning  equipment 


^ Closed  ended  questions  may  need  to  be  pre-coded  to  facilitate  data 
entry. 

^ If  scanning  equipment  is  being  used  for  coding,  questionnaires  may  need 
to  be  formatted  to  accommodate  scanning  technology. 

Survey  instruments  will  need  to  be  printed  (introduction  letters, 
questionnaires,  instruction  sheets,  self-addressed  stamped  envelopes, 
etc.) 

Questionnaires  may  need  to  be  pre-coded  with  identification  numbers  or 
printed  on  various  colored  paper  to  identify  respondents  or  sub-sets  of 
respondents. 

Sampling  protocols  should  be  implemented  to  ensure  randomized 
samples. 

^ Contact  labels  (for  envelopes)  may  need  to  be  developed  from  the 
sample  frame  and  put  on  envelopes. 

Envelopes  will  need  appropriate  postage. 

^ Survey  instruments  may  need  to  be  put  into  envelopes  for  distribution. 

An  alternative  to  the  above  tasks,  a mail  distribution  house  might  be 
contracted  to  organize  and  disseminate  survey  instruments  to 
respondents. 
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training  program  may 
take  3 to  6 hours  to 
adequately  instruct 
Interviewers  on  the 
tasks  and 
responsibilities.. 


— 

Administer 


Telephone  surveys 

^ Questionnaires  may  be  pre-coded  and  printed  (if  data  recording  is  paper- 
based).  Alternatively,  questionnaires  may  be  programmed  for  use  in  a 
computer-aided  telephone  interviewing  system  (a  computer  software 
program  that  enables  simultaneous  interviewing  and  data  entry). 

^ Call  record  sheets  may  be  printed  using  data  from  the  sample  frame  (see 
an  example  in  Appendix  C).  Another  option  is  to  use  a call  management 
software  program  (which  may  be  part  of  the  computer-aided  telephone 
interviewing  system).  In  either  case,  sampling  protocols  should  be 
implemented  to  ensure  representative  samples  (if  probability  samples 
are  being  used). 

^ Train  telephone  interviewers  for  survey  administration.  A basic  training 
program  might  include: 

■ Instruction  on  interviewing  protocols  (techniques,  building 
relationships,  recording  data,  ethics,  etc.) 

Interviewers  need  to  establish  a friendly  relationship  with 
respondents,  gain  cooperation  and  trust. 

Introduction  and  screening  is  very  important  in  building  the 
relationship  between  interviewer  and  respondent. 
Interviewers  should  approach  each  interview  as  though  it 
were  to  take  place  immediately. 

^ Answer  respondents'  questions. 

Interviewers  should  always  be  pleasant  and  professional  in 
interviewing  situation. 

Vocal  expressions  should  possess  clear  enunciation, 
moderate  rate  of  speech,  low  pitch  and  inflection. 

Interviewers  should: 

■ Ask  every  question  exactly  as  worded  and 
structured  in  the  questionnaire. 

• Ask  questions  in  a positive  manner. 

■ Repeat  and  clarify  questions  that  are  misunderstood 
or  misinterpreted. 

■ Probe  for  clarification  of  responses 

■ Listen  to  full  answers  provided  by  respondents. 

^ Interviewers  should  not: 

■ Suggest  answers  to  a respondent 

■ Ask  leading  questions  when  probing  or  clarifying 
responses. 

■ Provide  personal  information. 

■ Offer  personal  opinions  about  survey  issues. 

■ Brief  interviewers  on  the  background  of  the  project. 

■ Brief  interviewers  on  survey  instruments  and  record  call  sheets 

Go  through  each  question  with  interviewers.  Ensure  they 
understand  each  question,  instructions  for  question,  skip 
patterns. 

Explain  proper  data  recording  procedures. 

Go  through  record  call  sheets  to  ensure  interviewers  and 
familiar  with  the  process. 

■ Have  interviewers  practice  interviewing  (such  as  interviewing 
each  other)  with  the  survey  instruments. 
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• Web-based  surveys 


Research  reveals  that 
respondents  are 
more  likely  to 
participate  in  a multi- 
page web-based 
survey  than  a single 
page  web-based 
survey  that  requires 
them  to  scroll  down 
the  page. 


Closed  ended  questions  may  need  to  be  pre-coded  to  facilitate  software 
programming. 

Questionnaires  will  need  to  be  programmed  into  a survey  computer 
software  program.  The  software  program  may  be  web-page  or  e-mail 
based,  or  an  electronic  file  attached  to  an  e-mail  message.  (Note:  this 
step  in  the  process  may  require  specialized  computer  software  or 
expertise). 

Sampling  protocols  should  be  implemented  to  ensure  randomized 
samples. 

Invitations  will  need  to  be  sent  to  sample  that  is  derived  from  the  sample 
program. 

In-person  surveys. 


^ Questionnaires  may  need  to  be  pre-coded  and  printed  (if  data  recording 
is  paper-based).  Alternatively,  questionnaires  may  need  to  be 
programmed  for  use  on  a hand  held  device  or  laptop  computer  for 
computer-aided  telephone  interviewing. 

Contact  record  sheets  and  protocols  will  need  to  be  organized  to  guide 
interviewers  in  the  selection  of  respondents. 

^ Train  interviewers  for  survey  administration.  The  basic  training  program 
is  presented  above  (telephone  survey). 
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n Collect  data  from  respondents 

Survey  instruments  (or  invitations  to  participate)  should  be  sent  to  respondents  in 

self-administered  surveys.  In  interviewer  assisted  surveys,  interviewers  should 

begin  the  interview  process  (within  a day  or  two  of  their  training  session). 

During  data  collection,  the  following  points  must  be  considered: 

• Respondents  should  always  be  given  a contact,  if  they  request,  in  case  they 
have  questions  or  concerns  they  wish  to  express.  Respondents  should  be 
able  to  check,  without  difficulty,  the  identity  and  validity  of  the  organizations 
or  individuals  contacting  them  for  the  survey  project. 

• Respondents  should  always  be  informed  that  the  survey  is  voluntary  and  that 
they  do  not  need  to  give  a reason  for  declining  participation.  They  should  not 
be  misled  when  being  asked  for  their  cooperation. 

• Respondents  should  be  given  an  accurate  estimate  of  interview  length. 

• The  survey  needs  to  clearly  state  how  the  results  will  be  used  and  reported. 

• The  respondents  must  be  told  how  the  issue  of  confidentiality  will  be  handled 
and  whether  or  not  their  answers  can  be  identified  in  the  results. 

In  self-administered  surveys,  respondents  should  be  provided  with 
information  that  will  enable  them  to  complete  the  questionnaire. 

^ In  interviewer-administered  surveys,  interviewers  should  have  training  to 
properly  administer  questionnaires  to  respondents  through  telephone 
interviews. 

Supervisors  should  monitor  flow  rates  (completion  rates  per  hour)  to 
control  costs  and  ensure  interviewers  are  maintaining  efficient 
completion  rates. 

• Quality  control  measures  should  be  established  and  employed,  such  as: 

• Appropriate  sample  control  procedures  should  be  organized  and 
implemented  for  all  data  collection  operations.  These  procedures  may 
track  the  completion  status  of  surveys,  monitor  quotas,  establish  flow 
rates,  etc. 

• Effective  control  systems  should  be  established  to  ensure  the  security  of 
data  transition  and  handling. 

• Interviews  should  be  monitored  or  confirmed  (i.e.  least  10%  of  completed 
interviews  should  be  confirmed  for  telephone  and  in-person  surveys). 

• 100%  data  entry  verification  should  be  conducted  for  collection  methods 
that  do  not  employ  computer-aided  data  entry  (e.g.  computer-aided 
telephone  interviewing). 
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d Prepare  data  for  analysis 


authorities  contract 
out  the  data  capture, 
cleaning  and  editing 
and  tabulation 
components  of  the 
survey  project. 

V J 


The  satisfaction  survey  data  will  need  to  be  cleaned  and  edited  in  preparation  for 
analysis.  The  cleaning  and  editing  process  involves  reviewing  questionnaires  to 
increase  accuracy  and  precision.  In  a review,  researchers  might  identify 
illegible,  incomplete,  inconsistent  or  ambiguous  responses.  As  well,  re-coding  of 
data  may  be  necessary  to  further  enhance  the  use  of  the  data. 

A codebook  should  be  established  to  ensure  internal  consistencies  within  the 
dataset.  Codebooks  can  be  very  helpful  if  more  than  one  individual  will  be 
involved  in  the  data  cleaning  and  editing.  For  instance,  a codebook  can  be  used 
to  train  data  entry  staff.  Codebooks  contain  instructions  and  necessary 
information  about  the  data  set.  It  is  common  for  every  question  (variable)  to  have 
descriptions  of  what  might  be  contained  in  responses  to  the  question.  For 
example,  a codebook  might  have  the  following  description  for  a question 


variable  name  - satisfaction  with  education 

question  number  - Question  3 

response  label  - 1 =very  satisfied 

2=satisfied 
3=dissatisfied 
4=very  dissatisfied 
5=don't  know 

Instructions  - Recode  label  5 as  missing  for  data 

analysis 


Researchers  should  run  a set  of  frequencies  to  revel  the  number  of  responses  for 
each  question  and  response  category.  A review  of  these  frequencies  will  give 
the  researcher  a rough  check  for  completeness  and  accuracy  of  the  data  (e.g. 
responses  should  not  exceed  the  total  number  of  respondents,  responses  should 
not  show  be  out  of  range  values,  etc.). 

Data  cleaning  involves  checks  of  data  consistency  and  missing  responses. 
Consistency  checks  identify  data  that  are  out  of  range,  logically  inconsistent  or 
have  extreme  values.  These  types  of  errors  are  inadmissible  in  the  data  set  and 
must  be  corrected  (i.e.  re-contacting  respondents)  or  discarded. 

Re-coding  (variable  transformation)  involves  transforming  data  to  create  new 
variables  or  modify  existing  variables.  This  process  is  common  with  open-ended 
questions  where  responses  are  re-coded  into  consistent  categories. 

Once  a researcher  is  confident  that  the  data  is  properly  cleaned  and  edited,  the 
data  set  is  ready  for  analysis. 
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G Analyze  data 

Data  analysis  is  the  process  of  transforming  raw  data  into  useable  information. 
The  basic  steps  of  the  analytic  process  consist  of  examining  the  issue  and  asking 
meaningful  questions,  and  developing  support  for  answers  that  are 
communicated  to  decision  makers  and  other  readers.  To  be  an  effective  support 
for  decision  making,  the  data  must  be  analyzed  appropriately.  The  analysis 
should: 

• Examine  the  issue  and  ask  meaningful  questions 

Generally,  follow  guidelines  for  analysis  accepted  in  the  field  of  survey 
research. 

^ Effective  data  analysis  typically  focuses  on  issue,  theme,  and  idea 
categorization,  rather  than  simply  presenting  the  survey  data. 

Apply  statistical  significance  tests  to  address  hypotheses,  where 
appropriate. 

^ Caution  should  be  observed  in  drawing  conclusions  concerning 
causality.  In  the  absence  of  certainty  that  a specific  cause  is  the  only 
one  consistent  with  facts,  cite  all  possible  explanations,  not  just  one. 

• Develop  support  for  answers 

^ Use  frequencies,  cross  tabulations  and  statistical  tests  to  identify 

proportional  representations  (percentages)  and  correlations  among  data. 
^ Indicate  the  rationale  for  the  selection  of  any  significance  tests  used. 
Indicate  details  about  any  transformations  (use  of  Z-Scores,  for  example) 
made  on  the  data. 

^ Outline  any  conclusions  drawn  from  the  analysis  or  limitations  of  the 
analysis. 

• Various  types  of  statistical  analyses  may  be  applied  to  the  data  to  reveal 
issues,  trends,  associations,  etc. 

^ Descriptions  of  central  tendencies  reveal  typical,  average  or 

representative  values  of  the  data  set  - mean  (average),  median  (middle 
value  - half  the  data  are  larger,  the  other  half  smaller),  and  mode  (most 
frequently  occurring  value). 

Other  descriptive  statistics  such  as  frequencies,  percentiles  and 
percentages  summarize  the  data  by  revealing  distributions  of  responses. 
In  satisfaction  surveys,  it  is  common  to  identify  the  percentage  of 
respondents  who  express  satisfaction  (e.g.  very  satisfied  or  satisfied). 
Cross-tabulations  examine  the  relationships  among  two  or  more 
variables.  For  example,  different  schools  against  satisfaction  levels). 
Statistical  significance  tests  reveal  notable  differences  within  the  data  set 
(Chi-square,  z scores,  t-tests,  analysis  of  variance,  etc.) 

^ Multivariate  analysis  is  conducted  to  determine  relationships  and 

associations  between  and  among  various  variables  in  the  data  set  (e.g. 
factor  analysis,  regression  analysis,  etc.) 
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• An  analytical  tool  commonly  used  in  satisfaction  research  is  an 

importance/satisfaction  matrix.  This  analysis  involves  plotting  ratings  from 
satisfaction  and  importance  scaled-response  questions  on  a grid  and  helps 
define  priorities  for  service  delivery.  The  grid  enables  researchers  to  visually 
identify  areas  for  service  improvement. 

Service  Improvement  Priority  Matrix 
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HIGH 


Quadrant  1 

Quadrant  2 

Concentrate  Here 

Keep  Up  the  Good  Work 

- Important  to  stakeholders 

- Important  to  stakeholders 

- Poor  performance 

- Good  performance 

3 - 

- Expectations  not  being  met 

- Expectations  being  met 

Quadrant  3 

Quadrant  4 

Low  Priority 

Possibie  Overkill 

2 

- Less  important  to  stakeholders 

- Less  important  to  stakeholders 

- Poor  performance 

- Good  performance 

- Not  priorities  for  improvement 

- Exceeding  expectations 

- May  present  opportunity  to 

re-allocate  resources 

I 2 

3 ^ 

LOW 

HIGH 

Satisfaction  Ratings 

(Average  Response) 


A school  authority  might  ask  two  scaled-response  questions  with  15 
items  related  to  delivery  of  education  services.  The  first  question  asks 
respondents  to  answer  using  an  importance  scale,  the  second  question  a 
satisfaction  scale  (Note;  if  an  item  is  asked  in  terms  of  importance,  it  is 
also  asked  in  the  context  of  satisfaction. 

Average  scores  for  importance  and  satisfaction  ratings  plot  the  mid-point 
on  each  axis  of  the  matrix.  The  researcher  then  plots  the  data  in  the 
matrix.  Those  items  that  fall  in  the  Quadrant  1 should  be  given  further 
consideration.  Those  that  land  in  Quadrant  3 are  considered  low  priority. 
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Communicating  Survey  Resuits 


n Identify  stakeholder  groups  that  will  receive  survey  results 

< 

Several  school 
authorities  present 
survey  results  to 
Parent  Councils. 

V y 

^ What  audiences  will  be  reviewing  and  using  the  report  (administration, 
teachers,  Alberta  Learning,  the  general  public,  etc.)? 

Will  readers  of  the  report  understand  survey  concepts  and  terminology? 

^ Will  audiences  expect  to  read  interpretation  of  the  data  in  the  report? 

What  key  themes  arise  from  the  data  results  (a  good  report  typically  conveys 
several  key  themes  from  the  data)? 

n Determine  methods  to  report  survey  results 

Reporting  may  employ  multiple  forms  (e.g.  paper,  electronic,  visual,  etc.)  and 
address  the  needs  of  different  audiences  and  purposes.  It  is  important  to 
consider  the  reporting  expectations  of  interested  individuals  or  groups  when 
deciding  which  method  is  appropriate. 

Survey  results  can  be  presented  in  various  forms,  such  as: 


The  documentation  of  survey  results  serves  as  a record  of  what  was  done  during 
the  survey  research  to  provide  a context  for  effective  and  informed  use  of  the 
results.  Reporting  should  provide  a complete,  unambiguous  and  multi-purpose 
record  of  the  survey,  including  the  data  produced  by  the  survey. 

Knowing  the  intended  audience  for  the  report  can  influence  the  way  a satisfaction 
survey  report  is  prepared.  Some  issues  that  researchers  should  consider  when 
preparing  a report,  include: 


• Oral  reports  are  often  used  to  inform  interested  individuals  or  groups  of 
survey  results.  Typically,  oral  reports  are  accompanied  by  a visual 
presentation  (using  an  overhead  projector  or  computer  and  LCD  projector) 
that  shows  graphical  illustrations  of  the  survey  findings. 

• An  executive  summary  might  identify  key  points  in  narrative  that  come  out 
of  a satisfaction  survey  (a  one  or  two  page  briefing). 

• A data  report  presents  tables  of  response  counts  and  proportions  for  all 
questions  asked  in  the  satisfaction  survey.  There  may  be  little  narrative 
presented  in  the  report,  other  than  an  introduction  that  provides  the  survey 
objectives  and  methodology. 

• A descriptive  or  narrative  report  usually  summarizes  and  explains  the 
survey  findings,  provides  interpretation  and  describes  various  analyzes,  and 
identifies  key  themes  arising  from  the  data. 
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Data  from  the  survey  can  also  be  reported  in  a variety  of  ways,  for  example; 

• Tables 


Satisfaction  with  Education 

Response 

Choice 

% of  Respondents 

All  respondents 
(n=1,200) 

Grade  4 
Students 
(n=400) 

Grade  8 
Students 
(n=400) 

Grade  1 1 
Students 
(n=400) 

Very  satisfied 

52 

55 

53 

50 

Satisfied 

32 

29 

32 

35 

Dissatisfied 

10 

11 

9 

8 

Very  Dissatisfied 

6 

5 

6 

7 

Total 

100 

100 

100 

100 

(Note:  the  n=  shows  the  number  of  respondents  in  each  category) 


• Charts 


Verysatisfied  Satisfied  Dissatisfied  Verydissatisfied 

%of  Respondents 


□ All  respondents  (n=1 ,200)  ■ Grade  4 Students  (n=400) 

□ Grade  8 Students  (n=400)  □ Grade  1 1 Students  (n=400) 
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n Prepare  survey  results  report 

To  enable  a clearer  understanding  of  the  results,  survey  reporting  should 
(particularly  in  a descriptive  or  narrative  report): 

• Identify  the  survey  objectives 

• Describe  methodologies 

A description  of  the  survey  population  (students,  parents,  teachers,  etc.). 
Details  of  the  sampling  method,  size,  and  frame. 

When  technically  relevant,  a statement  of  response  rates  and  a 
discussion  of  any  possible  bias  due  to  non-response. 

The  precision  (sampling  error)  and  confidence  levels. 

The  timeframe  in  which  the  survey  was  conducted. 

^ The  method(s)  by  which  the  data  was  collected. 

• Describe  respondents 

A demographic  profile  of  respondents  should  be  presented  in  the  report. 
Comparisons  of  the  sample  with  the  target  population  might  be  made  to 
determine  whether  the  sample  is  representative. 

• Present  survey  results 

^ The  relevant  factual  findings  obtained.  The  level  of  detail  provided  in  the 
report  will  depend  on  the  intended  audience,  the  medium  of 
dissemination,  and  the  intended  use  of  the  survey  data. 

Comparison  of  actual  results  to  past  performance  where  possible. 

Use  graphs/figures  in  addition  to  text  or  tables  to  communicate 
messages. 

^ In  presenting  rounded  data,  use  the  number  of  significant  digits  that  is 
the  maximum  number  consistent  with  the  utility  of  the  data  (e.g.  4.6% 
rather  than  4.57869%). 

^ The  draft  report  should  be  checked  and  any  errors  corrected  (e.g. 
consistency  of  figures  used  in  the  text,  tables  and  charts,  verification  of 
accuracy  of  external  data,  and  simple  arithmetic). 

^4  Conclusions  presented  in  the  report  should  be  consistent  with  the 
evidence  available  in  the  survey  data. 
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• Other  reporting  issues 

Provide  description  of  limitations  in  interpreting  the  results. 

When  aggregating  and  releasing  data  outside  the  organization,  data  cells 
with  counts  of  less  than  six  should  be  suppressed.  Although,  in  some 
circumstances,  this  number  may  be  as  high  as  ten  where  the  information 
could  be  used  to  identify  an  individual  respondent  or  small  group  of 
respondents.  Sensitivity  of  data  cell  counts  is  usually  determined  by  the 
need  to  protect  individual  characteristics  of  respondents.  Methods 
typically  used  to  transform  data  to  protect  personal  information  include: 

• Collapsing  Categories  - Data  are  grouped  (or  re-coded)  into  cell 
categories  so  that  none  of  the  cells  are  considered  sensitive. 

• Cell  Suppression  - Sensitive  cells  are  deleted  from  a table.  It  should 
be  noted,  however,  that  it  may  be  possible  to  obtain  the  value  of  the 
suppressed  cell  by  solving  a system  of  linear  equations  based  on 
other  cells  in  a table. 

• Rounding  - The  data  cell  value  is  rounded  to  a number  that  would 
reasonably  protect  personal  information  of  respondents  (e.g.  5 or 
10). 

• Stripping  - The  removal  of  any  names  and  other  personal  identifiers 
from  records,  while  leaving  other  information  such  as  opinion  data  in 
the  records. 

^ Organize  and  document  references. 

Include  copies  of  the  survey  instruments  (and  other  relevant  technical 
data)  in  the  appendices. 

The  report  should  be  subject  to  extensive  review  by  researchers, 
decision  makers,  and  other  relevant  staff,  as  appropriate,  to  ensure 
quality  and  readability.  Reports  should  be  edited  meticulously. 
y^  An  communications  expert  (inhouse  or  agency)  might  be  consulted  to 
review  for  sensitivity  of  issues  and  plain-language  and  to  determine  if  a 
plan  or  strategy  is  needed  to  distribute  results  to  external  audiences. 
Presentations  may  be  required  to  School  Board  Members,  Principals, 
Stakeholders,  etc. 

Reporting  and  disclosure  of  personal  information  (e.g.  names  of  respondents, 
individual  personal  characteristics,  etc.)  must  comply  with  the  privacy  protection 
provisions  of  the  Freedom  of  information  and  Protection  of  Privacy  Act  and  the 
FOIP  Regulation  (see  Other  Issues  Related  to  Satisfaction  Surveys). 
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n Communicate  results  to  interested  individuals  or  groups 

The  key  messages  communicated  to  interested  individuals  and  groups  should 
reflect  the  primary  findings  of  the  survey  and  address  the  information  needs  of 
applicable  audiences. 

Those  who  have  been  engaged  in  survey  research  are  encouraged  to  share 
survey  results  with  internal  staff,  administrators  and  boards,  and  external 
stakeholders  and  interested  parties. 

In  some  cases,  data  may  be  sent  to  Alberta  Learning  as  required  in  the 
Accountability  Framework  of  the  department. 


y 
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Implementing  Satisfaction  Survey  Results 


O Develop  and  implement  initiatives  to  address  survey  results 

The  survey  findings  may  provide  feedback  that  requires  education  providers  to 
develop  or  enhance  initiatives  to  address  new  priorities,  enhance  services,  meet 
expectations,  etc.  The  results  may  identify  what  works  well  and  what  may  need 
to  modified  or  started. 

The  survey  results  may  also  have  direct  relevance  and  influence  to  business  or 
education  plan  developed  by  school  authorities  and  schools. 


d Gain  additional  feedback  from  stakeholders 


authorities  use 
Parent  Councils  to 
explore  issues  drawn 
out  the  survey  in 
greater  detail. 

V y 


In  some  cases,  the  findings  of  satisfaction  surveys  may  need  further 
examination.  Additional  research  may  be  needed  to  clarify  information. 

There  are  various  research  techniques  that  may  be  undertaken  such  as  focus 
groups,  in-depth  interviews,  and  additional  surveys. 

Organizations  may  consider  further  research  opportunities  with  targeted 
populations  in  cases  where  new  issues  are  identified  from  the  survey  data. 


CJ  Evaluate  the  successes  and  challenges  of  the  survey  project 


Researchers  who  conduct  satisfaction  surveys  are  encouraged  to  assess  the 
effectiveness  and  value  of  the  research  project  and  provide  suggestions  for 
improving  the  project  or,  if  applicable,  recommend  the  need  for  additional 
research  or  waves  of  the  survey. 


• The  survey  project  and  its  results  may  be  evaluated  formally  or  informally.  In 
other  words,  sometimes  the  evaluation  will  be  thorough,  structured  and 
formal,  while  other  times  it  will  be  impressionistic,  subjective  and  informal. 
The  choice  of  process  will  depend  on  resources  and  interests  of  the 
organization  that  directs  the  survey  research. 

• The  basic  framework  for  a project  evaluation  is  to  determine  what  went  well, 
and  what  did  not.  Researchers  might  examine  the  survey  management, 
methodologies  or  results  to  identify  strengths  and  weaknesses. 

• Solicit  opinions  from  decision  makers,  stakeholders  and  partners  in  the 
evaluation.  Request  feedback  on  how  survey  design  and  administration  can 
be  improved. 
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n Plan  future  satisfaction  measurement  issues 

The  evaluation  may  provide  conclusions  or  assess  areas  for  improvement  for 
future  satisfaction  survey  projects.  The  following  approaches  highlight  the  results 
that  might  be  provided  from  the  project  evaluation: 

Assess  if  and  how  the  results  were  actually  used  for  decision  making. 

V"  Render  a judgment  on  the  value  of  the  survey  project. 

Assist  decision  makers  in  determining  whether  to  conduct  the  survey 
research  project  again  in  the  future. 

Identify  areas  for  improvement. 


y 

y 
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other  Issues  Related  to  Conducing  Satisfaction  Surveys 

Surveying  Children  and  Youth 

Special  care,  attention  and  precautions  should  be  taken  when  conducting  survey 
research  with  children  and  youth. 

School  authorities  and  schools  within  Alberta's  learning  system  have  policies  and 
procedures  (formal  or  informal)  related  to  survey  research  projects  involving  their 
learners.  Schools  or  school  boards  who  are  planning  to  survey  children  or  youth 
in  Alberta's  learning  system  should  gain  proper  authorization  within  the  policies 
and  procedures  of  their  jurisdictions.  Organizations  who  are  conducting  surveys 
with  students  are  encouraged  to  contact  policy  administrators  early  in  the  survey 
planning  process  to  determine  the  requirements  and  specifications  needed  to 
proceed  with  a satisfaction  survey. 

Alberta  Learning  has  established  the  following  guidelines  for  conducting  survey 
research  with  students  within  the  learning  system  {Survey  Research  Policy, 
Guidelines  and  Best  Practices,  2002): 

• When  a child  in  elementary  school  is  a respondent,  parents  or  guardians 
must  be  informed  of  the  survey  research  project  before  conducting  an 
interview  (interviewer-administered  or  self-administered  interviews).  ^ 

• When  a youth  in  a junior  or  senior  high  is  a respondent,  a parent,  guardian  or 
relevant  adult  (such  as  a teacher)  must  be  informed  of  the  project. 

• In  any  other  environment  such  as  the  home,  school  yard  or  other  public 
places,  the  child's  or  youth's  parent  or  guardian  must  provide  consent  before 
the  child/youth  is  approached  for  an  interview. 

• Consent  by  a responsible  adult  should  not  be  interpreted  as  constituting 
permission,  as  the  child/youth  must  be  granted  an  opportunity  to  accept  or 
decline  his/her  participation  in  the  interview.  A child/youth  must  not  be 
approached,  under  any  circumstances,  unless  the  child/youth  is 


Section  33  (2)  of  the  Freedom  of  Information  and  Protection  of  Privacy  Act  imposes  a duty  on  public  bodies 
(government  departments,  agencies,  boards,  commissions,  etc.)  to  ensure  that  individuals  are  properly  informed  about 
the  purposes  for  which  their  personal  information  will  be  used.  In  1999,  Alberta's  Information  and  Privacy  Commissioner 
proposed  that  elementary  school  children  may  not  fully  comprehend  the  purposes  for  which  their  personal  information  will 
be  used  or  raise  critical  questions  about  providing  personal  situations.  As  such,  the  Commissioner  recommended  that  "in 
situations  involving  the  collection  of  personal  information  from  elementary  school  children,  it  is  recommended  the  Public 
Body  inform  parents/guardians,  in  writing  of  the  following: 

• The  purpose(s)  for  which  the  information  will  be  used. 

• The  legal  authority  for  the  collection. 

• The  title  address  and  phone  number  of  a person  who  can  answer  questions. 

• How  the  child's  information  will  be  used. 

• The  organizations  involved  in  the  study. 

• The  measures  in  place  to  ensure  protection  of  personal  information,  and 

• That  participation  in  the  survey  is  voluntary." 

Clark,  Robert  C.,  Information  and  Privacy  Commissioner.  Investigation  Report  99-IR-002.  Alberta:  1999. 
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accompanied  by  his/her  parent,  guardian  or  a relevant  adult  (such  as  a 
teacher). 

• When  requesting  permission  to  interview  a child/youth,  sufficient  information 
must  be  given  to  the  parent,  guardian  or  a relevant  adult  (such  as  a teacher) 
for  him/her  to  give  adequate  consideration  to  the  decision  about  granting 
permission  for  the  interview.  The  types  of  information  to  provide  parents, 
guardians  or  a relevant  adult  are  covered  in  Section  33(2)  of  the  Freedom  of 
Information  and  Protection  of  Privacy  Act. 

• It  is  desirable  to  have  the  parent,  guardian  or  a relevant  adult  (such  as  a 
teacher)  close  at  hand  while  the  interview  is  being  conducted. 

• While  it  may  be  imperative  to  avoid  certain  subjects  when  interviewing 
children  (e.g.  a topic  that  might  frighten  a child),  a similar  research  subject 
may  be  covered  with  youth  if  appropriate  precautions  are  taken.  Research 
topics  that  may  need  special  care  or  precautions  include: 

• Issues  that  could  upset  or  worry  the  child/youth. 

• Those  that  risk  creating  tension  between  the  child/youth  and  his/her 
parents. 

• Those  relating  to  potentially  sensitive  family  situations  (e.g.  parental 
relationships,  income,  use  of  stimulants,  and  family  illnesses). 

• Those  relating  to  race,  religion,  or  similar  socially  or  politically  sensitive 
matters. 

• Those  concerned  with  sexual  activities. 

• Those  related  to  illegal  or  socially  unacceptable  activities. 

• The  overall  welfare  and  well-being  of  the  child/youth  participating  in  survey 
projects  should  be  given  utmost  consideration.  In  all  cases,  the  safety,  rights 
and  interests  of  the  child  or  youth  must  be  advocated  and  upheld.  All  survey 
research  carried  out  with  children  or  youth  must  be  conducted  to  high  ethical 
standards  so  that  no  abuse,  real  or  perceived,  is  caused  to  the  child  or  youth 
involved. 
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Freedom  of  Information  and  Protection  of  Privacy  Specifications 

Satisfaction  surveys  involve  the  collection,  use,  retention,  disclosure  and 
disposition  of  personal  information.  Part  2 of  the  Freedom  of  Information  and 
Protection  of  Privacy  Act  (FOlPAct)  and  the  FOIP  Regulation  have  established 
privacy  protection  provisions  for  how  organizations  can  gather,  use  or  disclose 
personal  information. 

Researchers  should  refer  to  the  following  documents  in  the  planning  phase  of  the 
satisfaction  survey  project  to  ensure  that  the  survey  adheres  to  FOIP  protocols: 

FOIP:  Conducting  Surveys:  A Guide  to  Privacy  Protection,  August  2003. 

FOIP:  Contract  Manager's  Guide,  December  2003. 

The  Information  Management,  Access  and  Privacy  Branch  of  Alberta 
Government  Services  has  prepared  a 'Best  Practices  Checklist'  to  assist 
government  staff  in  addressing  privacy  compliance  challenges  as  it  relates  to 
survey  research.  This  Checklist  is  presented  in  Appendix  I for  consideration. 
Jurisdictions  who  are  engaged  in  survey  research  are  encouraged  to  review  the 
criteria  presented  in  the  'Checklist'  to  ensure  the  survey  projects  comply  with  the 
FOlPAct.  Staff  are  also  encouraged  to  consult  with  the  Department's  FOIP 
Coordinator  to  seek  advice  on  privacy  and  compliance  issues. 
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Definition  of  Terms 

The  following  are  definitions  of  terms  commonly  used  in  survey  research,  and 

presented  in  this  document: 

• Anonymity  - Information  about  respondents  must  be  reported  in  ways  that 
do  not  permit  the  identification  of  any  individual. 

• Bias  - The  tendency,  during  any  step  in  a survey,  to  systematically  favor  or 
give  advantage  to  answers  or  findings  which  will  cause  resulting  estimates  to 
deviate  in  one  direction  from  the  true  value.  Bias  may  or  may  not  be 
intentional. 

• Census  - A survey  in  which  the  researcher  attempts  to  gather  data  from  all 
members  of  a population. 

• Coding  - A process  of  converting  questionnaire  information  to  numbers  or 
symbols  to  facilitate  subsequent  data  processing  and  analysis. 

• Codebook  - A set  of  question  responses  and  their  associated  values  (code 
numbers).  Typically,  codebooks  are  used  to  document  categories  or  values 
assigned  to  question  responses  on  a survey  questionnaire. 

• Computer-assisted  telephone  interviewing  (CA  Tl)  - A type  of  telephone 
interviewing  in  which  interviewers  enter  responses  to  questions  into  a 
computer  as  they  are  received.  A computer  viewing  screen  automatically 
displays  questions  that  interviewers  ask  of  respondents  through  a telephone. 

• Confidentiality  - The  situation  where  the  privacy  of  information  provided  by 
individual  respondents  to  a survey  is  maintained  and  the  information  about 
individual  respondents  cannot  be  derived  from  the  published  survey  results. 

• Consent  - Respondents  (or  appropriate  parents/guardians)  providing 
permission  or  approval  to  be  interviewed.  Respondents  have  the  right  to 
terminate  interviews  at  any  time  and  to  withhold  any  information  they  choose. 

• Data  - Collective  reference  to  individual  items  of  information. 

• Data  Cleaning  - The  application  of  procedures  of  coding  and  identifying 
missing,  invalid  or  inconsistent  entries. 

• Interview  - Any  form  of  direct  or  indirect  contact  with  respondents  where  the 
object  is  to  acquire  data  or  information  which  could  be  used  in  whole  or  in 
part  for  the  purposes  of  a survey  research  project. 

• Open  Competition  - Service  requirements  are  advertised  to  the  general 
public  and  all  interested  parties  are  invited  to  submit  a proposal  (e.g.  through 
the  MERX  system  or  through  an  advertisement  in  provincial  newspapers). 
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• Personal  Information  - Relates  to  data  or  information  about  an  identifiable 
individual.  It  includes  an  individual's  name,  address,  telephone  number,  age, 
gender,  marital  status,  educational  and  employment  history,  and  personal 
opinions  and  views. ^ A complete  definition  is  available  in  the  FOlPAct, 
section  1 (n). 

• Pilot  Test  - A small  scale  survey,  using  respondents  from  the  target 
population  for  the  purpose  of  testing  the  integrated  functioning  of  all 
component  parts  of  the  survey  operation.  Revisions  can  then  be  made  as 
necessary  before  the  full-scale  survey  is  undertaken. 

• Precision  - A measure  of  the  closeness  of  the  sample  estimates  to  the  result 
from  a census  taken  under  identical  conditions. 

• Pre-Test  - A preliminary  testing  of  individual  component  parts  of  a survey  to 
check  that  each  component  functions  as  planned.  Each  component  can  then 
be  revised  as  needed. 

• Probability  Sample  - Any  method  of  selection  of  units  from  a population  in 
which: 

• every  unit  has  a known  and  calculable  chance  (greater  than 
zero)  of  being  selected, 

• the  sample  is  drawn  by  a random  selection  procedure,  and 

• the  probabilities  of  selection  are  used  to  project  the  results  from 
the  sample  to  the  population. 

• Questionnaire  - A series  of  structured  questions  designed  to  elicit 
information  on  one  or  more  topics  from  a respondent. 

• Rating  scale  - A type  of  survey  question  designed  to  record  direction  and 
strength  of  a respondent's  perception  toward  a specific  topic. 

• Reliability  - The  extent  to  which  a survey,  if  repeated  using  another  (but 
statistically  equivalent)  sample  and  identical  questionnaire  and  procedures, 
would  produce  the  same  results. 

• Representative  sample  surveys  - Surveys  in  which  the  sample  is  a 
selection  from  a larger  population  having  the  essential  characteristics  of  the 
total  population. 

• Request  for  Proposals  (RFP)  - A document  specifying  the  requirements  of 
the  survey  project  that  is  sent  out  to  contractors.  The  contractors  then  reply 
(if  interested)  with  proposals  based  on  these  requirements. 


3 

Freedom  of  Information  and  Protection  of  Privacy,  Conducting  Surveys:  A Guide  to  Privacy  and  Protection,  ISBN  0- 
7785-1799-3,  Government  of  Alberta,  September  2001. 


Definition  of  Terms 


-42- 


A Practical  Guide  to  Conducting  Surveys  within  Alberta’s  Basic  Learning  System 


• Respondent  - Any  individual  or  organization  from  whom  any  information  is 
sought  for  the  purposes  of  a survey  research  project.  The  term  covers  cases 
where  information  is  to  be  obtained  by  verbal  interviewing  techniques,  postal 
and  other  self-completion  questionnaires,  mechanical  or  electronic 
equipment,  observation  and  any  other  method  where  the  identity  of  the 
provider  of  information  may  be  recorded  or  otherwise  traceable. 

• Rotations  - The  manner  in  which  various  questions  or  items  within  a 
question  are  asked  or  shown  in  different  order  for  each  interview.  This 
process  helps  to  eliminate  order  bias  that  might  develop  if  questions  were 
asked  or  shown  in  exactly  the  same  order  for  each  interview  conducted  in  a 
particular  survey. 

• Sample  frame  - Any  list,  material,  or  device  that  identifies,  and  allows 
access  to  members  of  a target  population. 

• Screening  - The  process  of  checking  whether  an  individual  or  a situation 
should  be  included  in  a survey  or  survey  question. 

• Selected  Competition  - A process  where  specified  contractors  are  invited  to 
respond  to  a Request  for  Proposal. 

• Single  Source  Selection  ■ A process  where  one  contractor  is  invited  to 
propose  on  the  provision  of  services. 

• Skips  - A device  used  in  questionnaires  to  guide  respondents  (or 
interviewers)  past  a (set  of)  question(s)  that  do  not  apply  to  a particular 
respondent. 

• Survey  - The  term  is  used  to  refer  to  the  general  method  of  data  gathering, 
wherein  a number  of  respondents  are  asked  identical  questions  through  a 
systematic  questionnaire  or  interview.  Instruments  used  to  collect  actual 
data  will  be  referred  to  as  'questionnaire  (s)'  or  'survey  instrument  (s).' 

• Survey  Instrument  - Any  device  used  to  solicit  or  gather  data  or  information 
from  a respondent,  for  example,  introduction  letters,  questionnaires, 
computers,  tape  recorders,  or  video  tape  machines. 

• Target  Population  - The  complete  group  of  units  to  which  survey  results  are 
to  apply.  These  units  may  be  persons,  animals,  objects,  businesses,  trips, 
etc. 

• Validity  - The  degree  to  which  a method  of  measurement  succeeds  in 
measuring  what  it  is  intended  to  measure. 

• Wave  - A set  of  activities  in  a survey  or  series  of  surveys  conducted  among 
the  same  population  using  the  same  methodology. 
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APPENDIX  A - Templates  for  Evaluation  of  Survey  Consultants 


Project  Name: 


Date  of  Review: 


This  form  provides  a framework  for  evaluating  proposals  submitted  by  outside  contractors  for  survey  research. 

The  evaluation  process  involves  assigning  rating  scores  (1  being  'does  not  meet  requirements'  and  5 being  'exceeds 
requirements')  to  various  criteria  depending  on  how  well  contractors  proposals  meet  the  requirements  of  each  criterion. 
In  some  cases,  there  may  be  some  qualifying  criteria  required  for  a proposal  to  be  considered.  A score  of  1 (does  not 
meet  requirements)  or  2 (somewhat  below  requirements)  would  cause  the  immediate  disqualification  of  a proposal.  In 
order  to  expedite  what  can  be  a lengthy  process,  these  criteria  may  be  evaluated  in  isolation.  Note  that  the  form  begins 
with  space  for  these  criteria. 

Each  criterion  can  be  weighted  to  reflect  the  relative  importance  of  the  requirements  to  the  survey  research  project  (e.g. 
one  criterion  may  be  more  important  to  the  project  than  another).  Assign  a weight  to  each  criterion  (below).  The  sum 
of  all  the  weights  should  equal  20.  Then  record  the  weight  for  each  category  in  the  space  provided  in  the  table,  and 
assign  a score  for  each  criterion.  You  may  want  to  include  specific  comments  (strengths  and  weaknesses)  regarding  the 
scores  given. 

After  scoring  each  category,  multiply  the  scores  by  the  assigned  weights.  Add  the  weighted  scores  and  record 
the  total  in  the  box  for  the  Overall  Weighted  Score. 


When  scoring,  use  the  following  scale: 

1 - Does  not  meet  requirements 

2 - Somewhat  below  requirements 

3 - Adequately  meets  requirements 

4 - Meets  requirements  very  well 

5 - Exceeds  requirements 


Assign  weights  to  categories: 

Understanding  of  Project  & Requirements 

Approach/Methodology 

Project  Timelines  & Cost 

Experience  & Stature 

Ability  to  Complete  the  Project 

Overall  Assessment 


Weights: 


Mandatory 

Criteria 

Comments 

Evaluation 

Strengths 

Weaknesses 

Score 

(1  or  2 causes  immediate  disqualiflcation) 

1 . Mandatory 
Criteria 

Criteria 

Comments 

Evaluation 

Strengths 

Weaknesses 

Weighted 

Score  X Weight  = Score 

2.  Understanding 
of  Project  & 
Requirements 

3.  Approach/ 
Methodology 

4.  Project 
Timelines  & 
Costs 

5.  Experience  & 
Stature 

6.  Ability  to 
Complete  the 
Project 

7.  Overall 
Assessment 

8.  Additional 
Criterion 

Overall  Weighted  Score: 
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The  following  summaries  present  information  that  can  be  used  to  evaluate  survey 
research  proposals  submitted  by  outside  contractors: 

1.  Understanding  of  Project  & Requirements 

• General  understanding 

• Is  the  methodology  appropriate  for  the  project  requirements? 

• Does  the  proposal  demonstrate  understanding  of  the  essential  aspects  of  the 
project? 

■ Will  the  end  report  address  the  objectives  of  the  project  based  on  the  proposal? 

• Detailed  understanding 

■ Does  the  proposal  demonstrate  understanding  of  the  issues  involved? 

■ Can  the  firm  anticipate  potential  problems? 

- What  solutions  does  the  firm  propose? 

2.  Approach/Methodoloqy 

• Methodological  reliability 

■ Does  the  proposal  provide  a high  probability  of  accomplishing  the  study 
objectives? 

• Quality  of  data 

■ Is  the  accuracy  of  the  data  ensured? 

• Statistical  precision 

■ Is  the  reliability  of  the  data  ensured? 

• Technical  integrity 

■ Is  the  Approach/Methodology  based  on  sound  research  design  principles? 

• Flexibility  of  design 

■ Is  there  enough  flexibility  to  ensure  success  if  project  parameters  change? 

3.  Project  Timelines  & Costs 

• Budget 

■ How  does  the  cost  compare  to  the  budget?  Is  it  reasonable? 

■ Have  the  components  of  the  budget  been  explained  clearly? 

• Timelines 

■ Are  the  proposed  timelines  agreeable  with  the  project  requirements? 

■ Have  the  components  of  the  timeline  been  explained  clearly? 

4.  Experience  & Stature 

• Education/Technical  expertise 

■ How  much  overall  experience  does  the  personnel/firm  have  in  data  collection  and 
data  analysis/report  writing? 

■ Does  the  firm  possess  sufficient  background  knowledge  for  the  project? 

• Experience 

• Does  the  firm  have  experience  in  similar  and/or  related  projects? 
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• Credibility 

■ Strong  references  from  previous  clients? 

5.  Ability  to  Complete  the  Project 

• Resources 

■ Does  the  firm  have/have  access  to  sufficient  resources  to  complete  the  project? 

■ Who  are  the  key  personnel  to  be  involved  in  the  project,  and  how  qualified  are 
they?  (Need  to  ensure  that  qualified  personnel  identified  in  the  proposal  will 
actually  be  the  ones  who  carry  out  the  survey  project) 

■ How  involved  are  key  personnel  in  each  phase? 

• Management/Supervisory  resources 

• Does  the  firm  demonstrate  an  ability  to  manage  the  various  aspects  of  the 
project,  and  have  they  allocated  sufficient/appropriate  resources  for  project 
management  and  supervision? 

6.  Overall  Assessment 


• Overall  quality 

■ Was  adequate  care,  attention  to  detail  and  effort  put  into  planning/creation  of  the 
proposal? 


• Style 

■ Clear/concise  writing:  carefully  edited;  well  presented  concepts;  attractive  and 
easily  understood  layout? 

• Involvement 

■ Have  provisions  been  identified  to  keep  the  clients  involved?  How/how  much? 

• Strategy 

■ Does  the  proposal  provide  a superior  strategy  for  accomplishing  goals  of  the 
project? 

■ Does  the  firm  add  extra  value  to  the  project? 

7.  Additional  Criterion 


• Issues  or  specifications  customized  to  the  project. 

8.  Additional  Criterion 


• Issues  or  specifications  customized  to  the  project. 

9.  Additional  Criterion 


• Issues  or  specifications  customized  to  the  project. 
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Project  Name: 

Name  of  Company/Firm:_ 
Name  of  Reviewer: 


Satisfaction  Survey 


ABC  Research  and  Consulting 
Jane  Smith 


_Date  of  Review:  February  1.  2004 


This  form  provides  a framework  for  evaluating  proposals  submitted  by  outside  contractors  for  survey  research. 

The  evaluation  process  involves  assigning  rating  scores  (1  being  'does  not  meet  requirements'  and  5 being  'exceeds 
requirements')  to  various  criteria  depending  on  how  well  contractors  proposals  meet  the  requirements  of  each  criterion. 
In  some  cases,  there  may  be  some  qualifying  criteria  required  for  a proposal  to  be  considered.  A score  of  1 (does  not 
meet  requirements)  or  2 (somewhat  below  requirements)  would  cause  the  immediate  disqualification  of  a proposal.  In 
order  to  expedite  what  can  be  a lengthy  process,  these  criteria  may  be  evaluated  in  isolation.  Note  that  the  form  begins 
with  space  for  these  criteria. 

Each  criterion  can  be  weighted  to  reflect  the  relative  importance  of  the  requirements  to  the  survey  research  project  (e.g. 
one  criterion  may  be  more  important  to  the  project  than  another).  Assign  a weight  to  each  criterion  (below).  The  sum 
of  all  the  weights  should  equal  20.  Then  record  the  weight  for  each  category  in  the  space  provided  in  the  table,  and 
assign  a score  for  each  criterion.  You  may  want  to  include  specific  comments  (strengths  and  weaknesses)  regarding  the 
scores  given. 

After  scoring  each  category,  multiply  the  scores  by  the  assigned  weights.  Add  the  weighted  scores  and  record 
the  total  in  the  box  for  the  Overall  Weighted  Score. 


When  scoring, use  the  following  scale: 

1 - Does  not  meet  requirements 

2 - Somewhat  below  requirements 

3 - Adequately  meets  requirements 

4 - Meets  requirements  very  well 

5 - Exceeds  requirements 


Assign  weights  to  categories:  Weights: 

Understanding  of  Project  & Requirements  5_ 

Approach/Methodology  3_ 

Project  Timelines  & Cost  4_ 

Experience  & Stature  2_ 

Ability  to  Complete  the  Project  4 

Overall  Assessment  2 


Mandatory 

Criteria 

Comments 

Evaluation 

Strengths 

Weaknesses 

Score 

(1  or  2 causes  immediate  disqualification) 

Bid  is  within  Budget 

Budget  = $12,000 
Bid  = $11,275 

5 

Criteria 

Comments 

Evaluation 

Strengths 

Weaknesses 

Weighted 

Score  X Weight  = Score 

Understanding  of 
Project  & 
Requirements 

Telephone  survey 
proposed,  matches 
objective  of  validity 

Limited  depth  of  interviewing 
in  telephone  survey 

3 

5 

15 

Approach/ 

Methodology 

Quality  control  measures 
documented 

Non-standard  quality  control 
measures 

4 

3 

12 

Project  Timelines  & 
Costs 

Well  within  budget 
boundary 

Timelines  are  not  detailed, 
may  be  some  risk  of  late 
delivery  of  results 

3 

4 

12 

Experience  & 
Stature 

Firm  is  well-known  in  this 
market 

Firm  has  not  completed 
comparable  project 

3 

2 

8 

Ability  to  Complete 
the  Project 

Firm  has  large  professional 
team 

Finn  has  larger  client  that 
may  distract  resources  during 
our  project 

2 

4 

8 

Overall  Assessment 

Firm  principals  present 
well,  understand  project 

Some  concerns  about 
importance  of  other  clients 

3 

2 

6 

Additional  Criterion 

None 

Overall  Weighted  Score: 

61 
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APPENDIX  B - Scale  Response  Questions 

The  following  four  step  process  is  offered  to  researchers  to  assist  in  the 

development  of  response  scale  questions: 

STEP  1:  Develop  a distinct  list  of  items  to  be  measured. 

n Determine  and  clarify  the  concepts,  ideas  or  theories  to  be  measured. 

n Determine  the  level  of  specificity  or  generality  required  from  the  results  to 
address  the  business  need  being  addressed. 

□ Determine  whether  items  of  concepts,  ideas  or  theories  being  measured  are 
distinct  (otherwise  respondents  may  experience  challenges  in  responding  to 
the  questions). 

□ The  items  in  the  list  should  not  be  ambiguous. 

n Avoid  lengthy  wording  that  compromises  clarity  (though  not  at  the  expense  of 
meaning). 

n Avoid  multiple  negatives. 

□ Avoid  double-barreled  items  where  the  items  convey  2 or  more  ideas. 

□ Avoid  ambiguous  pronoun  references. 

□ Use  both  positively  and  negatively  worded  items  to  avoid  bias  where 
respondents  are  inclined  to  affirm  items  regardless  of  their  content  (however, 
there  may  be  a trade-off  between  avoiding  bias  and  creating  confusion). 
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STEP  2:  Determine  the  measurement  format. 


□ Measurement  formats  (e.g.  satisfaction,  agreement,  interested,  etc.)  should 
be  compatible  with  the  items  generated  for  the  question. 

□ The  following  presents  aspects  of  scale  portions  that  researchers  should 
consider  when  deciding  upon  an  appropriate  measurement  format: 

Number  of  categories  - The  number  of  categories  or  options  selected  by  the 
researcher  should  provide  sufficient  discrimination  within  the  scale.  In  other 
words,  the  higher  the  number  of  categories  or  options,  the  finer  the 
distinctions  available  for  analysis.  However,  shorter  scales  have  the 
advantage  that  they  are  less  of  a burden  on  respondents.  It  is  important  for 
researchers  to  consider  whether  respondents  will  find  the  distinctions 
relevant  and  meaningful  (e.g.  can  the  respondent  actually  distinguish  a 
difference  between  somewhat  agree  and  slightly  agree?). 


4-Point  Scale 

• strongly  agree 

• Agree 

• Disagree 

• Strongly  disagree 


7 -Point  Scale 

• strongly  agree 

• Somewhat  agree 

• Slightly  agree 

• Neither  agree  nor  disagree 

• Slightly  disagree 

• Somewhat  disagree 

• Strongly  disagree 


Balanced  or  unbalanced  scales  - The  researcher  must  decide  whether  the 
scale  should  be  balanced  or  unbalanced.  The  balanced  scale  provides  an 
equal  number  of  response  categories  on  both  ends  of  the  continuum.  This  is 
the  most  common  form  of  rating  and  generally  satisfies  the  requirements  for 
interval  measurement.  Conversely,  an  unbalanced  scale  is  used  when  the 
direction  of  response  is  generally  known  and  finer  distinctions  on  one  end  of 
the  continuum  are  desired.  Although  unbalanced  scales  are  sometimes  used 
in  social  and  marketing  research,  they  are  generally  not  recommended 
because  the  question  has  the  potential  to  bias  the  respondent  and  it  may  be 
difficult  to  analyze  and  interpret  the  data. 


Balanced  Scale 

• strongly  agree 

• Somewhat  agree 

• Neither  agree  nor  disagree 

• Somewhat  disagree 

• Strongly  disagree 


Unbalanced  Scale 

• Very  strongly  agree 

• Strongly  agree 

• Somewhat  agree 

• Somewhat  disagree 

• Disagree 
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Odd  or  Even  Number  of  Categories  - When  balanced  scales  are  used, 
researchers  must  determine  whether  the  scale  will  have  an  odd  or  even 
number  of  categories  (the  use  of  odd  number  of  categories  typically 
designates  the  middle  category  as  'neutral'  such  as  neither  agree  nor 
disagree,  except  in  the  application  of  an  unbalanced  scale).  The  decision  to 
use  either  odd  or  even  number  of  response  categories  usually  depends  on 
the  researcher's  assumptions  about  respondents'  mindset.  Researchers  who 
advocate  even-numbered  categories  generally  propose  that  respondents 
may  use  the  neutral  options  to  hide  their  opinion  and  they  should  be  forced  to 
indicate  some  degree  of  opinion.  Advocates  of  odd-numbered  categories 
suggest  that  respondents  can  be  neutral  in  their  opinion  or  perspectives  and, 
thus,  should  be  allowed  to  state  their  ambivalence. 


4-Point  Scale 

• strongly  agree 

• Agree 

• Disagree 

• Strongly  disagree 


5-Point  Scale 

• strongly  agree 

• Somewhat  agree 

• Neither  agree  nor  disagree 

• Somewhat  disagree 

• Disagree 


The  following  are  general  considerations  for  using  odd  or  even  number  of 
categories: 


■ Even  Number  of  Categories  (such  as  a 4 point  scale)  - Is  appropriate  in 
situations  where  a 'forced'  choice  is  necessary.  However,  it  should  be 
noted  that  forced-choice  may  increase  non-response  rate. 

■ Odd  Number  of  Categories  (such  as  a 5 point  scale)  - Is  appropriate  if  a 
neutral  opinion  is  an  appropriate  response.  But,  the  researcher  may 
have  difficulties  interpreting  the  neutral  proportion  of  responses. 

STEP  3:  Have  question  items  and  response  scales  reviewed  by  survey 
committee  members  who  have  survey  expertise. 


□ Issues  that  should  be  considered  when  questions  and  response  scales  are 
being  reviewed,  include: 

• Relevance  of  each  item  and  what  it  is  intended  to  measure. 

• Clarity  and  conciseness  of  items. 

• The  potential  for  response  bias. 


STEP  4:  Evaluate  response  scale  questions  in  a survey  pre-test. 

□ De-brief  interviewers  or  pilot-test  respondents  to  determine  whether  items 
are  clear,  understandable,  and  distinct  in  meaning. 

□ Review  the  performance  of  items  to  assess  variances  in  responses  (high 
variance  of  scale  items  is  desirable). 

□ If  there  are  concerns  about  a response  scale  question,  try  alternatives  in  the 
pre-test. 
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APPENDIX  C • Call  Record  Management  Form 


Interviewer  # 

Date: 

Total 

(Project  Name) 

B - Busy 

Final 

Conrtplete 

NA  - No  Answer 

Phone  # 

Cain 

Cali  2 

Calls 

Status 

Number 

AM  - Answprino 

555-1535 

Machine 

555-9812 

CB  - Arranged 

Cali  Back 

555-9383 

555-8175 

555-2978 

C - Completed 

555-8304 

R - Refused 

555-5678 

Fax  - Fax 

555-3497 

1 - Incomplete 

555-0987 

T - Terminated 

555-6821 

NQ  - Not  Qualified 

555-4093 

NIS-Not  in  Service 

555-0112 

Bus  - Business  # 

555-7891 

L-Language 

555-8374 

Total  Calls  made 

555-5930 

555-2947 

NOTES: 

555-4930 

555-0081 

555-4581 

555-7623 

555-6611 

555-9641 

555-1398 

555-7549 

555-4760 

555-3322 

555-8861 

555-2875 

555-6830 

555-1881 
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APPENDIX  D - FOIP  Best  Practices  Checklist 


(Source;  Freedom  of  Information  and  Protection  of  Privacy:  Conducting  Surveys:  A Guide 
to  Privacy  Protection,  ISBN  0-7785-2097-8,  Government  of  Alberta,  revised  August  2003) 


Planning  and  Design 

• Clearly  define  the  issues  you  wish  to  address  through  survey  research.  This  will  help  limit 
collection  of  information  to  that  which  is  necessary  to  address  the  issues. 


• if  the  survey  is  going  to  gather  large  amounts  of  personal  information,  very  sensitive 
personal  information  and/or  retain  personal  information  for  a length  of  time,  a privacy 
impact  assessment  shouid  be  performed. 


• Ensure  staff  has  a clear  understanding  of  the  privacy  issues  before  beginning  survey 
research. 


• If  using  an  external  contractor  for  any  stage  of  the  project,  have  a written  agreement  or 
contract  in  place  ensuring  compliance  with  the  FOIP  Act. 


• Whenever  possible,  design  the  survey  so  that  no  personal  information  is  collected. 


• If  the  survey  cannot  be  carried  out  anonymously,  design  it  so  that  personal  information  is 
transformed  before  use  or  disclosure. 


• If  using  coded  surveys,  ensure  that  procedures  are  in  piace  to  minimize  the  extent  of 
access  to  both  sets  of  data. 


• Make  sure  the  survey  participants  are  informed  of  the  purpose  of  the  survey  and  how  you 
will  be  using  any  personal  information  that  may  be  transformed. 

Sample  Selection 

• When  you  know  in  advance  that  client  information  will  be  used  to  select  a survey  sample, 
provide  notice  of  this  use  at  the  time  of  collection. 

• When  you  have  not  anticipated  use  of  personal  information  to  select  a survey  sample  at 
the  time  of  collection,  use  that  information  only  if  the  use  is  consistent  with  the  original 
purpose  of  collection  or  you  have  individual  written  consent. 


• If  you  are  asking  the  Information  and  Privacy  Commissioner  for  permission  to  collect 
personal  information,  complete  a privacy  impact  assessment  to  demonstrate  the  need. 


• Before  sharing  data  to  select  a survey  sample  make  sure  there  is  authority  to  collect  and 
disclose,  and  a personal  information  sharing  agreement  is  in  place. 

• If  possible,  avoid  indirect  collection  of  personal  information  to  obtain  a survey  research 
sample.  Instead,  have  the  public  body,  or  other  institution  that  maintains  the  personal 
information,  contact  potential  participants  directly  on  your  behalf. 
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• Before  sharing  data  or  contacting  potential  research  participants  on  behalf  of  another 
public  body,  you  should  ensure  that  you  have  the  authority  to  use  or  disclose  the 
personal  information  for  these  purposes  under  sections  39  or  40(1)  of  the  FOIP  Act. 


• When  contacting  potential  research  participants  on  behalf  of  another  public  body,  ensure 
that  replies  go  directly  to  the  public  body  conducting  the  survey. 


• Any  collection  of  personal  information  done  on  behalf  of  a public  body  requires  a notice  of 
collection  (see  Appendix  2). 

Data  Collection 

• If  personal  information  is  collected  for  a purpose  not  directly  related  to  the  survey,  keep 
the  two  types  of  information  separate  and  the  use  and  disclosure  of  this  information 
should  be  made  clear. 


• Ensure  you  have  the  authority  to  collect  the  personal  information  required  for  the  survey 
under  section  33  of  the  FOIP  Act. 


• Before  collecting  personal  information  indirectly  ensure  you  have  the  authority  to  do  so 
under  section  34(1)  or  (3)  of  the  FOIP  Act. 


• Limit  the  amount  of  personal  information  collected  to  what  is  strictly  necessary. 


• When  contacting  potential  survey  participants,  take  steps  to  protect  their  privacy  by  not 
disclosing  to  third  parties  the  name  of  your  institution  or  the  reason  for  contacting  the 
potential  survey  participants. 

• Unless  the  survey  is  done  anonymously,  provide  assurances  of  confidentiality  only  with 
the  proviso  that  disclosure  of  personal  information  may  occur  if  required  by  statute  or  the 
courts. 

• When  collecting  personal  information  to  conduct  a survey,  provide  notice  of  collection  in 
compliance  with  section  34(2)  of  the  FOIP  Act. 


• Provide  survey  participants  with  sufficient  information  about  the  survey  so  that  they 
understand  the  use  being  made  of  their  personal  information. 


• Whenever  possible,  collect  personal  information  directly  from  the  subject  individual. 


• Obtain  prior  written  consent  from  each  individual  if  the  intent  is  to  disclose  personal 
information  that  could  identify  him  or  her  to  those  the  survey  is  about. 

Data  Analysis 

• Use  and  disclose  personal  information  only  for  the  purposes  specified  to  the  survey 
participants  at  the  time  of  collection. 


• Before  using  personal  information  for  a purpose  not  specified  at  the  time  of  collection, 
obtain  the  individual’s  written  consent. 
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Reporting  Results 

• Report  survey  results  as  aggregate  information. 

• Do  not  report  results  of  small  cells  (i.e.  5 or  fewer  participants). 

• Consider  other  ways  of  transforming  personal  information  into  non-identifiable 
information. 

Records  Management 

• Whenever  possible  store  personal  information  separately  from  the  survey  responses. 

• Keep  a record  of  the  fact  that  a personal  information  bank  is  used  to  select  survey 
samples. 

• Ensure  that  you  have  a records  retention  and  disposition  schedule  in  place  for  all  records 
related  to  the  survey  and  follow  it. 
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APPENDIX  E - Error  Checks  for  Satisfaction  Surveys 


Error  Type 

Problem 

How  Error  may  be 
Addressed 

Sampling 

errors 

Target  population 
inadequately  defined 

Invalid  survey  results 

Re-defined  survey 
population 

Poor  sample  frame 

Potential  respondents 
will  be  missed  or 
inappropriate 
respondents  included 

Clean  and  update 
sample  frame 

Data 

collection 

errors 

Interviewer  errors: 

- failing  to  ask 
questions  properly 

- not  following 
questionnaire  flow 
-leading  respondents 

- improper  recording 
of  answers 

Inaccurate  or  poor  data 

- proper  selection, 
training  and 
supervision  of 
interviewers 

Respondent 

errors 

Respondent  provide 
incorrect  information 

Invalid  data 

- ensure  questions 
are  worded  and 
formatted  properly 

Respondent  lacks 
required  information 

Question  omitted  or 
guessed  at 

- allow  for  uncertainty 
in  question  answer 
(don't  know) 

- ensure  proper 
sample  frame  is  used 

Data 

administration 

errors 

Coding  errors 

Invalid  and  unreliable 
data 

- pre-code 
questionnaires 

- check  coding 

- train  coders 

Data  entry  errors 

invalid  and  unreliable 
data 

- use  computer 
software  programs 
customized  for  data 
capture 

-Train  data  entry  staff 

- Conduct  verification 
of  data  entry 
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